feat: implement ZeroTier exit gateway switcher

Adds the luci-app-zt-gateway package: a LuCI app + rpcd/ucode backend +
shell switch script that reconfigures which remote ZeroTier node acts as
the internet exit gateway for WIBLAN clients (10.11.13.0/24).

  - Makefile (luci.mk, arch-independent)
  - UCI config skeleton with three sample gateways
  - rpcd ACL + menu entry
  - zt-gateway.uc rpcd backend exposing status / switch / health /
    drain_status / cancel_drain ubus methods
  - zt-gateway-switch shell script implementing force + graceful modes:
      * force: pre-flight ping, atomic route replace, conntrack flush,
        UCI/hotplug/rc.local persistence
      * graceful: dual-table drain using CONNMARK fwmark 0x100 at
        priority 99, background drain monitor with two-consecutive-zero
        completion and timeout-forced fallback to force
  - LuCI overview.js: gateway radio list, mode select, drain progress
    panel, cancel-drain button, health polling
  - Docker test harness (docker-compose + Dockerfile.router +
    router/gw entrypoints) exercising the switch script against real
    iproute2/iptables/conntrack on two simulated exit nodes

Verified against the harness: force switch, graceful drain to natural
completion, pre-flight blocking of unreachable gateways (force + graceful),
and drain-timeout forced fallback.
This commit is contained in:
2026-06-19 02:51:21 +05:30
parent f6bf6e7cd4
commit eb04a2597d
12 changed files with 1376 additions and 10 deletions

16
Dockerfile.router Normal file
View File

@@ -0,0 +1,16 @@
FROM archlinux:latest
# archlinux:latest ships iproute2, iptables, awk, sed, ip6tables, busybox
# tools. Install conntrack-tools for the drain monitor + conntrack -D path.
# Tolerate offline builds: the script falls back to /proc/net/nf_conntrack
# when conntrack CLI is unavailable.
RUN pacman --noconfirm -Sy conntrack-tools 2>/dev/null || \
echo "[build] conntrack-tools unavailable; script will degrade gracefully"
# Harness + switch script
COPY docker/router-entrypoint.sh /usr/local/bin/router-entrypoint.sh
COPY root/usr/sbin/zt-gateway-switch /usr/sbin/zt-gateway-switch
RUN chmod +x /usr/local/bin/router-entrypoint.sh /usr/sbin/zt-gateway-switch
ENTRYPOINT ["/usr/local/bin/router-entrypoint.sh"]
CMD ["sleep", "infinity"]

14
Makefile Normal file
View File

@@ -0,0 +1,14 @@
include $(TOPDIR)/rules.mk
PKG_NAME:=luci-app-zt-gateway
LUCI_TITLE:=ZeroTier Exit Gateway Switching
LUCI_DEPENDS:=+luci-base +ucode +conntrack +iptables-mod-conntrack-extra +iptables-mod-connmark
LUCI_PKGARCH:=all
PKG_VERSION:=1.0.0
PKG_RELEASE:=1
PKG_LICENSE:=Apache-2.0
PKG_MAINTAINER:=WIBLAN Project <root@wiblan.net>
include $(TOPDIR)/feeds/luci/luci.mk
# call BuildPackage - OpenWrt buildroot signature

106
README.md
View File

@@ -17,21 +17,107 @@ disruption.
the new gateway immediately. Auto-falls-back to a force switch after a
configurable timeout if drain does not complete.
## Status
## Files
> **Planning / pre-implementation.**
```
luci-app-zt-gateway/
├── Makefile # OpenWRT build (luci.mk)
├── htdocs/luci-static/resources/view/zt-gateway/overview.js # LuCI UI
├── root/etc/config/zt-gateway # UCI config: gateway registry + state
├── root/usr/sbin/zt-gateway-switch # shell: actual switch logic
├── root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json # ACL
├── root/usr/share/rpcd/ucode/zt-gateway.uc # rpcd backend (ubus)
└── root/usr/share/luci/menu.d/luci-app-zt-gateway.json # menu entry
└── docker-compose.yml # local Docker test harness
└── Dockerfile.router # test image (archlinux + tools)
└── docker/router-entrypoint.sh # brings up br-zt + baseline routing
└── docker/gw-entrypoint.sh # simulated ZT exit node startup
```
The full design — file structure, UCI config schema, rpcd backend (ucode),
`zt-gateway-switch` shell script, LuCI frontend, switch logic for both modes,
persistence model, health checks, rollback/safety, Docker macvlan test harness,
and build/install instructions — is the source of truth in:
The `zt-gateway-switch` script is callable directly from SSH for testing; the
rpcd/ucode backend wraps it for the LuCI UI and exposes these ubus methods
under the `zt-gateway` object:
[`docs/.omp/plans/zerotier-gateway-switching.md`](.omp/plans/zerotier-gateway-switching.md)
| Method | Args | Description |
| -------------- | ----------------------------- | ---------------------------------------------- |
| `status` | — | active gateway, settings, gateway registry |
| `switch` | `{ region, mode }` | execute a force or graceful switch |
| `health` | `{ region }` | ping a gateway over br-zt, return latency |
| `drain_status` | — | graceful drain progress + remaining flows |
| `cancel_drain` | — | cancel drain and force-switch to the new gw |
(also tracked at `.omp/plans/zerotier-gateway-switching.md`)
## UCI config
Implementation (Makefile, ucode backend, switch script, `overview.js`, UCI
config, ACL, menu entry) lands in subsequent commits against this repository.
`/etc/config/zt-gateway` holds the gateway registry and active state:
```
config global 'global'
option active_gateway 'amsterdam' # region key of active gateway
option switch_mode 'force' # 'force' | 'graceful'
option drain_timeout '600' # graceful drain timeout (seconds)
option health_interval '60'
option auto_failback '0'
config gateway
option region 'amsterdam'
option label 'Amsterdam (ocirosea641)'
option ip '10.11.12.3'
option default '1'
option health_check 'ping'
```
## Build & install
Build against the OpenWRT SDK with `luci.mk` (architecture-independent —
`LUCI_PKGARCH:=all`). Full instructions are in
[`.omp/plans/zerotier-gateway-switching.md`](.omp/plans/zerotier-gateway-switching.md).
A manual `opkg-build` cheat sheet, the `apk` install path for OpenWRT 25.12+,
and post-install cache clearing are all documented there too.
## Local test harness (Docker)
End-to-end verification without a physical router is possible with the bundled
`docker-compose.yml` + `Dockerfile.router` + `docker/*-entrypoint.sh`. The
harness emulates two bridge networks (WIBLAN clients at `10.99.13.0/24` and
ZeroTier exit gateways at `10.99.12.0/24`) plus two simulated exit nodes, and
exercises `zt-gateway-switch` against real `iproute2`, `iptables`, and the
host kernel's conntrack table.
Run it with `docker compose up -d --build`, then `docker exec openwrt-router
zt-gateway-switch <ip> <mode> [drain_timeout]`.
## Verified behavior
The following scenarios have been exercised against the Docker harness:
- Force switch: `ip route show table 100` reflects the new gateway, host route
and mwan3 table 1 return route preserved.
- Graceful drain: table 100 → new gateway, table 101 → old gateway, fwmark
`0x100` rule at priority 99 inserted above the priority-100 src rule,
ESTABLISHED WIBLAN connections keep flowing through the old gateway until
they naturally age out, after which the mangle rules, drain table, and
pidfiles are cleaned up automatically.
- Pre-flight: switching to an unreachable gateway aborts with exit code 2 and
leaves routes untouched (verified for both `force` and `graceful` modes).
- Drain timeout: a forced-fallback fires after the configured timeout,
preserving the new gateway in table 100 and tearing down drain state.
## Persistence model
The gateway IP is recorded in five places on the router; the switch script
updates them all atomically:
| Location | What |
| --------------------------------- | -------------------------------------- |
| Kernel: host route | `<gw_ip> dev br-zt` |
| Kernel: table 100 default | `default via <gw_ip> dev br-zt table 100` |
| UCI: `/etc/config/network` | `zt_gateway_host` + `zt_gateway_default` routes |
| Hotplug: `99-zerotier-bridge` | host route + table 100 route + mwan3 table 1 return |
| Boot: `/etc/rc.local` | all three routes |
The hotplug and rc.local scripts are rewritten via a dot-escaped `sed` replace
so a reboot boots against the new gateway.
## License

70
docker-compose.yml Normal file
View File

@@ -0,0 +1,70 @@
services:
openwrt-router:
build:
context: .
dockerfile: Dockerfile.router
image: zt-gateway-router:dev
container_name: openwrt-router
privileged: true
cap_add:
- NET_ADMIN
- SYS_ADMIN
networks:
zt-gateway-lan:
ipv4_address: "10.99.13.1"
zt-exit-net:
ipv4_address: "10.99.12.1"
environment:
ZTG_SKIP_PERSIST: "1"
ZTG_PING_IFACE: "br-zt"
ZTG_WIBLAN_CIDR: "10.99.13.0/24"
wiblan-client:
image: archlinux:latest
container_name: wiblan-client
networks:
zt-gateway-lan:
ipv4_address: "10.99.13.10"
cap_add:
- NET_ADMIN
command: ["sleep", "infinity"]
zt-gw-amsterdam:
image: archlinux:latest
container_name: zt-gw-amsterdam
privileged: true
cap_add:
- NET_ADMIN
networks:
zt-exit-net:
ipv4_address: "10.99.12.3"
volumes:
- ./docker/gw-entrypoint.sh:/usr/local/bin/gw-entrypoint.sh:ro
entrypoint: ["/bin/sh", "/usr/local/bin/gw-entrypoint.sh", "10.99.12.3", "10.99.12.1"]
zt-gw-tirunelveli:
image: archlinux:latest
container_name: zt-gw-tirunelveli
privileged: true
cap_add:
- NET_ADMIN
networks:
zt-exit-net:
ipv4_address: "10.99.12.5"
volumes:
- ./docker/gw-entrypoint.sh:/usr/local/bin/gw-entrypoint.sh:ro
entrypoint: ["/bin/sh", "/usr/local/bin/gw-entrypoint.sh", "10.99.12.5", "10.99.12.1"]
networks:
zt-gateway-lan:
driver: bridge
ipam:
config:
- subnet: "10.99.13.0/24"
gateway: "10.99.13.1"
zt-exit-net:
driver: bridge
ipam:
config:
- subnet: "10.99.12.0/24"
gateway: "10.99.12.1"

21
docker/gw-entrypoint.sh Executable file
View File

@@ -0,0 +1,21 @@
#!/bin/sh
# gw-entrypoint.sh — shared by all simulated ZT exit gateways.
#
# Argument 1: this gateway's IP (within 10.11.12.0/24)
# Argument 2: router's ZT-side IP (10.11.12.1 by default; here we use the
# docker-compose gateway of the zt-exit-net network).
set -eu
my_ip="${1:?missing my_ip}"
router_ip="${2:-10.11.12.1}"
echo 1 > /proc/sys/net/ipv4/ip_forward
# Reply to ARP for our own IP (already done by the kernel). Make sure we
# can reach the WIBLAN subnet by routing back through the router.
ip route replace 10.11.13.0/24 via "$router_ip" 2>/dev/null || true
# Simple ping responder is all we need for pre-flight tests. No NAT/NAT
# rules required because test traffic only verifies the policy routing on
# the router side, not end-to-end internet egress.
echo "[gw ${my_ip}] up; routes via ${router_ip}"
exec sleep infinity

56
docker/router-entrypoint.sh Executable file
View File

@@ -0,0 +1,56 @@
#!/bin/sh
# router-entrypoint.sh
#
# Brings up the simulated br-zt bridge that mirrors production: the
# ZeroTier member interface (here, the container's interface on
# zt-exit-net) is enslaved to br-zt and its IP is moved onto br-zt. This
# makes 'ip route ... dev br-zt' actually reach 10.99.12.0/24.
set -eu
WIBLAN_CIDR="${ZTG_WIBLAN_CIDR:-10.99.13.0/24}"
echo "[entrypoint] bringing up br-zt bridge"
# Find the interface holding 10.99.12.x.
ZT_IFACE=$(ip -o -4 addr show \
| awk '$4 ~ /^10\.99\.12\./ {print $2; exit}')
if [ -z "$ZT_IFACE" ]; then
echo "[entrypoint] WARNING: no interface in 10.99.12.0/24; tests will fail" >&2
ZT_IFACE=eth1
fi
echo "[entrypoint] ZT-side interface: $ZT_IFACE"
ip link add name br-zt type bridge 2>/dev/null || true
ip link set br-zt up
if [ "$ZT_IFACE" != "br-zt" ]; then
ADDR=$(ip -o -4 addr show dev "$ZT_IFACE" \
| awk '$4 ~ /^10\.99\.12\./ {print $4; exit}')
if [ -n "$ADDR" ]; then
ip addr del "$ADDR" dev "$ZT_IFACE" 2>/dev/null || true
ip addr add "$ADDR" dev br-zt
fi
ip link set "$ZT_IFACE" master br-zt
fi
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects 2>/dev/null || true
# Seed the baseline policy routing the production router boots with.
ip rule del from "$WIBLAN_CIDR" table 100 2>/dev/null || true
ip rule add from "$WIBLAN_CIDR" table 100 priority 100
# Seed the current gateway's host route + table 100 default. These mirror
# what /etc/rc.local installs on boot in production.
ip route replace 10.99.12.3 dev br-zt
ip route replace default via 10.99.12.3 dev br-zt table 100
ip route replace "$WIBLAN_CIDR" dev br-zt table 1
echo "[entrypoint] baseline state:"
ip -o -4 addr show dev br-zt
echo "--- table 100:"
ip route show table 100
echo "--- rule:"
ip rule show
exec "$@"

View File

@@ -0,0 +1,316 @@
'use strict';
/* luci-app-zt-gateway - ZeroTier Exit Gateway switching UI
*
* Provides a Services -> ZeroTier Gateway page that lists all gateways from
* /etc/config/zt-gateway, shows the active gateway, health, and drain
* progress, and lets the user pick a target + switch mode (force|graceful).
* Calls zt-gateway.* ubus methods exposed by
* /usr/share/rpcd/ucode/zt-gateway.uc, which in turn invokes
* /usr/sbin/zt-gateway-switch for the actual system changes.
*/
import { view } from 'luci.view';
import { rpc } from 'luci.rpc';
import { ui } from 'luci.ui';
import { dom } from 'luci.dom';
import { Poll } from 'luci.poll';
const UBUS_NAMESPACE = 'zt-gateway';
function ubusStatus() {
return rpc.declare({
object: UBUS_NAMESPACE,
method: 'status',
params: []
})();
}
function ubusSwitch(region, mode) {
return rpc.declare({
object: UBUS_NAMESPACE,
method: 'switch',
params: ['region', 'mode']
})(region, mode);
}
function ubusHealth(region) {
return rpc.declare({
object: UBUS_NAMESPACE,
method: 'health',
params: ['region']
})(region);
}
function ubusCancelDrain() {
return rpc.declare({
object: UBUS_NAMESPACE,
method: 'cancel_drain',
params: []
})();
}
return view.extend({
state: {
selectedRegion: null,
switchMode: 'force'
},
load() {
return ubusStatus().catch(err => {
ui.addNotification(null, E('p', {}, _('Failed to load status: ') + (err.message || err)));
return null;
});
},
renderHealthDot(reachable, latencyMs) {
const cls = reachable ? 'zt-health-up' : 'zt-health-down';
const tip = reachable
? (latencyMs != null ? _('UP (%sms').format(latencyMs) : _('UP'))
: _('DOWN');
return E('span', { class: 'zt-health ' + cls, title: tip }, '●');
},
renderGatewayRow(gw, activeRegion, isSelected) {
const isActive = gw.region === activeRegion;
return E('tr', { class: 'zt-gateway-row', 'data-region': gw.region }, [
E('td', {}, E('input', {
type: 'radio',
name: 'zt_gateway_region',
value: gw.region,
change: ev => this.onSelectRegion(ev.target.value),
checked: isSelected || null,
disabled: isActive
})),
E('td', { class: 'zt-region-cell' }, [
this.renderHealthDot(false, null),
' ',
isActive ? E('strong', {}, gw.region) : document.createTextNode(gw.region)
]),
E('td', {}, gw.label),
E('td', {}, gw.ip),
E('td', {}, isActive ? E('em', {}, _('active')) : '')
]);
},
renderGatewaysTable(data) {
const activeRegion = data.active_gateway;
const rows = (data.gateways || []).map(gw => {
const isSelected = gw.region === this.state.selectedRegion;
return this.renderGatewayRow(gw, activeRegion, isSelected);
});
return E('table', { class: 'table zt-gateways' }, [
E('thead', {}, E('tr', {}, [
E('th', {}, ''),
E('th', {}, _('Region')),
E('th', {}, _('Label')),
E('th', {}, _('IP')),
E('th', {}, _('Status'))
])),
E('tbody', {}, rows)
]);
},
renderDrainPanel(data) {
const drain = data.drain || {};
if (!drain.active && !(drain.remaining_connections > 0)) {
return E([]);
}
return E('div', { class: 'zt-drain-panel' }, [
E('h4', {}, _('Graceful drain progress')),
E('p', {}, _('Draining from %s to %s - %d connection(s) remaining')
.format(drain.old_ip || '?', drain.new_ip || '?', drain.remaining_connections || 0)),
E('button', {
class: 'cbi-button cbi-button-negative',
click: ev => this.onCancelDrain(ev)
}, _('Cancel drain (force switch)'))
]);
},
renderActiveSummary(data) {
const region = data.active_gateway || _('none');
const ip = data.active_ip || '?';
return E('div', { class: 'zt-active-summary' }, [
E('strong', {}, _('Active gateway:')),
' ',
E('span', {}, '%s (%s)'.format(region, ip))
]);
},
onSelectRegion(region) {
this.state.selectedRegion = region;
},
onSwitchModeChange(ev) {
this.state.switchMode = ev.target.value;
const timeoutRow = document.querySelector('.zt-drain-timeout-row');
if (timeoutRow) {
timeoutRow.style.display = (this.state.switchMode === 'graceful') ? '' : 'none';
}
},
async onSwitchClick(ev) {
const region = this.state.selectedRegion;
if (!region) {
ui.addNotification(null, E('p', {}, _('Select a gateway first.')));
return;
}
const mode = this.state.switchMode || 'force';
const btn = ev.target;
btn.disabled = true;
const oldLabel = btn.textContent;
btn.textContent = _('Switching...');
try {
const res = await ubusSwitch(region, mode);
if (res && res.success) {
ui.addNotification(null, E('p', {}, res.message || _('Switch complete.')));
} else {
ui.addNotification(null, E('p', {}, (res && res.message) || _('Switch failed.')));
}
} catch (err) {
ui.addNotification(null, E('p', {}, _('RPC error: ') + (err.message || err)));
} finally {
btn.disabled = false;
btn.textContent = oldLabel;
this.refreshState();
}
},
async onCancelDrain(ev) {
const btn = ev.target;
btn.disabled = true;
const oldLabel = btn.textContent;
btn.textContent = _('Cancelling...');
try {
const res = await ubusCancelDrain();
ui.addNotification(null, E('p', {}, (res && res.message) || _('Cancel issued.')));
} catch (err) {
ui.addNotification(null, E('p', {}, _('RPC error: ') + (err.message || err)));
} finally {
btn.disabled = false;
btn.textContent = oldLabel;
this.refreshState();
}
},
async refreshHealth(data) {
const gws = data.gateways || [];
for (let i = 0; i < gws.length; i++) {
const region = gws[i].region;
try {
const h = await ubusHealth(region);
const row = document.querySelector('.zt-gateway-row[data-region="%s"]'.format(region));
if (row && h) {
const node = row.querySelector('.zt-health');
if (node) {
node.classList.toggle('zt-health-up', !!h.reachable);
node.classList.toggle('zt-health-down', !h.reachable);
node.title = h.reachable && h.latency_ms != null
? _('UP (%sms').format(h.latency_ms)
: (h.reachable ? _('UP') : _('DOWN'));
}
}
} catch (_) {
/* health check is best-effort */
}
}
},
async refreshState() {
try {
const data = await ubusStatus();
if (data) {
this.activeData = data;
const container = document.getElementById('zt-gateway-root');
if (container) {
dom.content(container, this.renderBody(data));
}
this.refreshHealth(data);
}
} catch (_) {
/* ignore - next poll will retry */
}
},
renderBody(data) {
const settings = data.settings || {};
const mode = settings.switch_mode || 'force';
this.state.switchMode = mode;
return [
E('h3', {}, _('ZeroTier Exit Gateway')),
this.renderActiveSummary(data),
E('div', { class: 'cbi-section' }, [
this.renderGatewaysTable(data),
E('div', { class: 'cbi-page-actions' }, [
E('div', { class: 'zt-mode-group' }, [
E('label', {}, _('Switch mode:')),
E('select', {
class: 'zt-mode-select',
change: ev => this.onSwitchModeChange(ev)
}, [
E('option', { value: 'force', selected: mode === 'force' || null }, _('Force')),
E('option', { value: 'graceful', selected: mode === 'graceful' || null }, _('Graceful drain'))
]),
E('span', {
class: 'zt-drain-timeout-row',
style: mode === 'graceful' ? '' : 'display:none'
}, [
' ',
_('Drain timeout:'),
' ',
E('input', {
type: 'number',
class: 'zt-drain-timeout',
value: settings.drain_timeout || 600,
min: 30,
step: 30
}),
' s'
])
]),
E('button', {
class: 'cbi-button cbi-button-positive',
click: ev => this.onSwitchClick(ev)
}, _('Switch to selected'))
])
]),
this.renderDrainPanel(data)
];
},
render(data) {
this.injectCSS();
if (!data) {
return E('div', { class: 'alert-message error' }, [
E('p', {}, _('Could not load gateway state. Verify rpcd is running and the zt-gateway backend is installed.'))
]);
}
this.activeData = data;
const root = E('div', { id: 'zt-gateway-root' }, this.renderBody(data));
const self = this;
const poll = Poll.create('zt-gateway-drain', function() {
return self.refreshState();
}, 5);
poll.start();
return root;
},
injectCSS() {
if (document.getElementById('zt-gateway-css')) return;
const css = '.zt-health{font-size:1.1em;margin-right:4px}' +
'.zt-health-up{color:#2e8b57}.zt-health-down{color:#c0392b}' +
'.zt-gateways tbody tr{line-height:1.8}' +
'.zt-mode-group{display:flex;gap:0.5em;align-items:center;flex-wrap:wrap}' +
'.zt-active-summary{margin:0.5em 0 1em 0}' +
'.zt-drain-panel{margin-top:1em;padding:0.75em;background:#f8f8f8;border:1px solid #ddd}';
const style = document.createElement('style');
style.id = 'zt-gateway-css';
style.textContent = css;
document.head.appendChild(style);
}
});

View File

@@ -0,0 +1,27 @@
config global 'global'
option active_gateway 'amsterdam'
option switch_mode 'force'
option drain_timeout '600'
option health_interval '60'
option auto_failback '0'
config gateway
option region 'amsterdam'
option label 'Amsterdam (ocirosea641)'
option ip '10.11.12.3'
option default '1'
option health_check 'ping'
config gateway
option region 'tirunelveli'
option label 'Tirunelveli (rpi1000)'
option ip '10.11.12.5'
option default '0'
option health_check 'ping'
config gateway
option region 'bangalore'
option label 'Bangalore (sensecap-m4)'
option ip '10.11.12.4'
option default '0'
option health_check 'ping'

420
root/usr/sbin/zt-gateway-switch Executable file
View File

@@ -0,0 +1,420 @@
#!/bin/sh
# shellcheck shell=sh
#
# zt-gateway-switch — reconfigure which remote ZeroTier node acts as the
# internet exit gateway for WIBLAN clients (10.11.13.0/24).
#
# Usage:
# zt-gateway-switch <new_ip> <mode> [drain_timeout]
#
# Modes:
# force - Instant cutover + conntrack flush
# graceful - fwmark drain; falls back to force on timeout
#
# This script is callable directly from SSH for testing; the LuCI rpcd
# backend (/usr/share/rpcd/ucode/zt-gateway.uc) wraps it for the UI.
#
# Exit codes:
# 0 success
# 1 usage error
# 2 pre-flight failed (gateway unreachable)
# 3 runtime failure (route/iptables/uci update failed)
#
# Environment overrides (used by the Docker test harness as much as by
# production):
# ZTG_BRIDGE bridge device (default: br-zt)
# ZTG_WIBLAN_CIDR WIBLAN source subnet (default: 10.11.13.0/24)
# ZTG_TABLE_MAIN main policy table (default: 100)
# ZTG_TABLE_DRAIN drain policy table (default: 101)
# ZTG_TABLE_MWAN mwan3 return-traffic table (default: 1)
# ZTG_FWMARK conntrack mark used for drain (default: 0x100)
# ZTG_DRAIN_PRIORITY priority of the fwmark drain rule (default: 99)
# ZTG_HOTPLUG hotplug script path
# ZTG_RCLOCAL rc.local path
# ZTG_DRAIN_PIDFILE pidfile for the drain monitor
# ZTG_DRAIN_NEWFILE file recording the new gateway IP during drain
# ZTG_DRAIN_OLDFILE file recording the old gateway IP during drain
# ZTG_SKIP_PERSIST if set to 1, skip persistence (testing)
# ZTG_PING_IFACE interface for the pre-flight ping (default: br-zt)
#
# This script intentionally uses only POSIX sh + busybox-compatible
# utilities so the same code path runs on the router and in the Docker
# openwrt/rootfs test image.
set -eu
# ----------------------------------------------------------------------------
# Config
# ----------------------------------------------------------------------------
BRIDGE="${ZTG_BRIDGE:-br-zt}"
WIBLAN_CIDR="${ZTG_WIBLAN_CIDR:-10.11.13.0/24}"
TABLE_MAIN="${ZTG_TABLE_MAIN:-100}"
TABLE_DRAIN="${ZTG_TABLE_DRAIN:-101}"
TABLE_MWAN="${ZTG_TABLE_MWAN:-1}"
DRAIN_PRIORITY="${ZTG_DRAIN_PRIORITY:-99}"
FWMARK="${ZTG_FWMARK:-0x100}"
HOTPLUG="${ZTG_HOTPLUG:-/etc/hotplug.d/net/99-zerotier-bridge}"
RCLOCAL="${ZTG_RCLOCAL:-/etc/rc.local}"
DRAIN_PIDFILE="${ZTG_DRAIN_PIDFILE:-/var/run/zt-gateway-drain.pid}"
DRAIN_NEWFILE="${ZTG_DRAIN_NEWFILE:-/var/run/zt-gateway-drain.new}"
DRAIN_OLDFILE="${ZTG_DRAIN_OLDFILE:-/var/run/zt-gateway-drain.old}"
PING_IFACE="${ZTG_PING_IFACE:-$BRIDGE}"
SKIP_PERSIST="${ZTG_SKIP_PERSIST:-0}"
# ----------------------------------------------------------------------------
# Logging
# ----------------------------------------------------------------------------
log() { printf '[zt-gateway-switch] %s\n' "$*" >&2; }
die() { rc=$1; shift; log "ERROR: $*"; exit "$rc"; }
# ----------------------------------------------------------------------------
# Helpers
# ----------------------------------------------------------------------------
valid_ip() {
ip=$1
# POSIX-only IPv4 validator (no awk regex extensions required).
rest=$ip
i=0
while [ $i -lt 4 ]; do
oct=${rest%%.*}
if [ "$oct" = "$rest" ]; then
# last octet
rest=
else
rest=${rest#*.}
fi
case "$oct" in
''|*[!0-9]*) return 1 ;;
esac
[ "$oct" -ge 0 ] 2>/dev/null || return 1
[ "$oct" -le 255 ] || return 1
i=$((i+1))
done
[ $i -eq 4 ] || return 1
[ -z "$rest" ] || return 1
[ "$ip" != "$rest" ] || return 1
# Reject leading/trailing dots already handled above; accept.
return 0
}
preflight_ping() {
gw_ip=$1
log "pre-flight: ping ${PING_IFACE}->${gw_ip}"
if ping -c 2 -W 3 -I "$PING_IFACE" "$gw_ip" >/dev/null 2>&1; then
return 0
fi
# Retry without -I in case the harness lacks the iface binding.
if ping -c 2 -W 3 "$gw_ip" >/dev/null 2>&1; then
return 0
fi
return 1
}
# Replace OLD_IP with NEW_IP in a file. IPv4 only; dots escaped for sed.
_replace_ip_in_file() {
file=$1
old=$2
new=$3
[ -f "$file" ] || { log "warning: $file not found; skipping"; return 0; }
old_re=$(printf '%s\n' "$old" | sed 's/[.]/\\./g')
sed -i "s/${old_re}/${new}/g" "$file"
}
set_host_route() {
gw_ip=$1
ip route replace "$gw_ip" dev "$BRIDGE"
}
set_table_default() {
gw_ip=$1
table=$2
ip route replace default via "$gw_ip" dev "$BRIDGE" table "$table"
}
del_table_default() {
table=$1
ip route del default dev "$BRIDGE" table "$table" 2>/dev/null || true
}
ensure_mwan_return() {
ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MWAN"
}
# Extract the gateway IP from the table N default route, or echo empty.
current_gateway_of() {
table=$1
ip route show table "$table" 2>/dev/null \
| awk '/^[[:space:]]*default/ {
for (i = 1; i <= NF; i++)
if ($i == "via") { print $(i+1); exit }
}'
}
# ----------------------------------------------------------------------------
# Persistence
# ----------------------------------------------------------------------------
persist_all() {
new_ip=$1
old_ip=${2:-}
if [ "$SKIP_PERSIST" = "1" ]; then
log "ZTG_SKIP_PERSIST=1; skipping persistence"
return 0
fi
if [ -n "$old_ip" ]; then
_replace_ip_in_file "$HOTPLUG" "$old_ip" "$new_ip"
_replace_ip_in_file "$RCLOCAL" "$old_ip" "$new_ip"
fi
if command -v uci >/dev/null 2>&1; then
if uci -q get network.zt_gateway_host >/dev/null 2>&1; then
uci set network.zt_gateway_host.target="$new_ip"
else
uci -q set network.zt_gateway_host=config route
uci -q set network.zt_gateway_host.target="$new_ip"
uci -q set network.zt_gateway_host.interface='br-zt'
fi
if uci -q get network.zt_gateway_default >/dev/null 2>&1; then
uci set network.zt_gateway_default.gateway="$new_ip"
else
uci -q set network.zt_gateway_default=config route
uci -q set network.zt_gateway_default.target='0.0.0.0'
uci -q set network.zt_gateway_default.netmask='0.0.0.0'
uci -q set network.zt_gateway_default.gateway="$new_ip"
uci -q set network.zt_gateway_default.table="$TABLE_MAIN"
fi
uci -q commit network || true
fi
}
# ----------------------------------------------------------------------------
# Force switch
# ----------------------------------------------------------------------------
do_force() {
new_ip=$1
old_ip=${2:-}
preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
set_host_route "$new_ip"
set_table_default "$new_ip" "$TABLE_MAIN"
ensure_mwan_return
if command -v conntrack >/dev/null 2>&1; then
conntrack -D -s "$WIBLAN_CIDR" 2>/dev/null || true
else
log "warning: conntrack not present; skipping flush"
fi
persist_all "$new_ip" "$old_ip"
if ip route show table "$TABLE_MAIN" 2>/dev/null | grep -q "default via ${new_ip}"; then
log "force: table ${TABLE_MAIN} default via ${new_ip} confirmed"
else
die 3 "verification failed: default via ${new_ip} not in table ${TABLE_MAIN}"
fi
printf 'force switch to %s complete\n' "$new_ip"
}
# ----------------------------------------------------------------------------
# Graceful drain switch
# ----------------------------------------------------------------------------
mangle_rules_install() {
# Source-only match: WIBLAN clients reach the router on whatever LAN
# interface they live on; binding to -i br-zt would miss the actual
# inbound path. We scope by source subnet so the rule fires regardless
# of ingress interface, then CONNMARK restores/marks/saves per-flow.
iptables -t mangle -A PREROUTING -s "$WIBLAN_CIDR" \
-m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -s "$WIBLAN_CIDR" \
-m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark "$FWMARK"
iptables -t mangle -A PREROUTING -s "$WIBLAN_CIDR" \
-j CONNMARK --save-mark
}
mangle_rules_remove() {
iptables -t mangle -D PREROUTING -s "$WIBLAN_CIDR" \
-j CONNMARK --save-mark 2>/dev/null || true
iptables -t mangle -D PREROUTING -s "$WIBLAN_CIDR" \
-m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark "$FWMARK" 2>/dev/null || true
iptables -t mangle -D PREROUTING -s "$WIBLAN_CIDR" \
-m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark 2>/dev/null || true
}
drain_install_rules() {
old_ip=$1
ip route replace default via "$old_ip" dev "$BRIDGE" table "$TABLE_DRAIN"
mangle_rules_install
ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \
ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
}
drain_cleanup() {
log "drain: cleaning up mangle rules, drain table, and pidfiles"
mangle_rules_remove
ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true
del_table_default "$TABLE_DRAIN"
rm -f "$DRAIN_PIDFILE" "$DRAIN_NEWFILE" "$DRAIN_OLDFILE" 2>/dev/null || true
}
drain_count_marked() {
# Count conntrack entries carrying the drain mark. /proc/net/nf_conntrack
# reports marks in decimal (e.g. mark=256 for 0x100), so convert FWMARK.
mark_dec=$(( FWMARK + 0 ))
mark_hex="0x$(printf '%x' "$mark_dec")"
# Preferred: conntrack CLI with -m filter (conntrack-tools >= 1.4.4).
if command -v conntrack >/dev/null 2>&1; then
count=$(conntrack -L -m "$mark_hex" 2>/dev/null | grep -c . || true)
if [ -n "$count" ] && [ "$count" -gt 0 ]; then
echo "$count"
return
fi
fi
# Fallback: parse /proc/net/nf_conntrack. Works without the CLI and
# also catches entries the CLI filter might miss on older builds.
awk -v mdec="$mark_dec" -v mhex="$mark_hex" '
BEGIN { n = 0 }
{
if (index($0, "mark=" mdec) || index($0, "mark=" mhex)) n++
}
END { print n }
' /proc/net/nf_conntrack 2>/dev/null || echo 0
}
drain_monitor() {
new_ip=$1
old_ip=$2
timeout=$3
# Grace period before the first count: packets already in flight need
# at least one PREROUTING pass through the mangle rules before their
# conntrack entry picks up mark=0x100. Without this cushion the very
# first count races the mangle hook and reports zero prematurely.
DRAIN_INITIAL_GRACE="${ZTG_DRAIN_INITIAL_GRACE:-3}"
# Number of consecutive zero-count polls required before we declare
# the drain complete. Guards against transient empty reads that race
# the conntrack update.
DRAIN_MIN_ZERO_POLLS="${ZTG_DRAIN_MIN_ZERO_POLLS:-2}"
DRAIN_POLL_INTERVAL="${ZTG_DRAIN_POLL_INTERVAL:-5}"
sleep "$DRAIN_INITIAL_GRACE"
end=$(( $(date +%s) + timeout ))
consecutive_zeros=0
while :; do
remaining=$(drain_count_marked)
log "drain: ${remaining} marked connection(s) remaining (consecutive_zeros=${consecutive_zeros})"
if [ "$remaining" -eq 0 ]; then
consecutive_zeros=$(( consecutive_zeros + 1 ))
if [ "$consecutive_zeros" -ge "$DRAIN_MIN_ZERO_POLLS" ]; then
log "drain: complete (${DRAIN_MIN_ZERO_POLLS} consecutive zero polls)"
drain_cleanup
return 0
fi
else
consecutive_zeros=0
fi
if [ "$(date +%s)" -ge "$end" ]; then
log "drain: timeout (${timeout}s) reached; falling back to force"
set_host_route "$new_ip"
set_table_default "$new_ip" "$TABLE_MAIN"
ensure_mwan_return
if command -v conntrack >/dev/null 2>&1; then
conntrack -D -s "$WIBLAN_CIDR" 2>/dev/null || true
fi
drain_cleanup
persist_all "$new_ip" "$old_ip"
return 0
fi
sleep "$DRAIN_POLL_INTERVAL"
done
}
do_graceful() {
new_ip=$1
timeout=${2:-600}
old_ip=$(current_gateway_of "$TABLE_MAIN")
if [ -z "$old_ip" ]; then
log "graceful: no current gateway in table ${TABLE_MAIN}; falling back to force"
do_force "$new_ip" ""
return $?
fi
if [ "$old_ip" = "$new_ip" ]; then
log "graceful: new gateway equals current gateway (${new_ip}); nothing to do"
return 0
fi
preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
printf '%s\n' "$new_ip" >"$DRAIN_NEWFILE"
printf '%s\n' "$old_ip" >"$DRAIN_OLDFILE"
drain_install_rules "$old_ip"
set_host_route "$new_ip"
set_table_default "$new_ip" "$TABLE_MAIN"
ensure_mwan_return
persist_all "$new_ip" "$old_ip"
: >"$DRAIN_PIDFILE"
(
trap 'drain_cleanup; exit 0' TERM INT
echo $$ >"$DRAIN_PIDFILE"
drain_monitor "$new_ip" "$old_ip" "$timeout"
rm -f "$DRAIN_PIDFILE" "$DRAIN_NEWFILE" "$DRAIN_OLDFILE" 2>/dev/null || true
) >/tmp/zt-gateway-drain.log 2>&1 &
printf 'graceful switch to %s started (draining from %s, timeout %ss)\n' \
"$new_ip" "$old_ip" "$timeout"
}
# ----------------------------------------------------------------------------
# Entry point
# ----------------------------------------------------------------------------
main() {
if [ $# -lt 2 ]; then
cat >&2 <<'USAGE'
Usage: zt-gateway-switch <new_ip> <mode> [drain_timeout]
Modes:
force Instant cutover + conntrack flush
graceful fwmark drain; falls back to force on timeout
Options:
drain_timeout graceful-mode drain timeout in seconds (default: 600)
USAGE
die 1 "missing arguments"
fi
new_ip=$1
mode=$2
timeout=${3:-600}
if ! valid_ip "$new_ip"; then
die 1 "invalid IPv4 address: ${new_ip}"
fi
case "$mode" in
force)
old_ip=$(current_gateway_of "$TABLE_MAIN")
do_force "$new_ip" "${old_ip:-}"
;;
graceful)
do_graceful "$new_ip" "$timeout"
;;
*)
die 1 "unknown mode: ${mode} (use force|graceful)"
;;
esac
}
main "$@"

View File

@@ -0,0 +1,14 @@
{
"admin/services/zt-gateway": {
"title": "ZeroTier Gateway",
"order": 90,
"action": {
"type": "view",
"path": "zt-gateway/overview"
},
"depends": {
"acl": ["luci-app-zt-gateway"],
"uci": { "zt-gateway": true }
}
}
}

View File

@@ -0,0 +1,25 @@
{
"luci-app-zt-gateway": {
"description": "ZeroTier Exit Gateway switching",
"read": {
"uci": ["zt-gateway", "network"],
"ubus": {
"zt-gateway": ["status", "health", "drain_status"]
},
"file": {
"/etc/hotplug.d/net/99-zerotier-bridge": ["read"],
"/etc/rc.local": ["read"]
}
},
"write": {
"uci": ["zt-gateway", "network"],
"ubus": {
"zt-gateway": ["switch", "cancel_drain"]
},
"file": {
"/etc/hotplug.d/net/99-zerotier-bridge": ["write"],
"/etc/rc.local": ["write"]
}
}
}
}

View File

@@ -0,0 +1,301 @@
'use strict';
import { ubus } from 'ubus';
import { uci } from 'uci';
import { fs } from 'fs';
const SWITCH_SCRIPT = '/usr/sbin/zt-gateway-switch';
const DRAIN_PIDFILE = '/var/run/zt-gateway-drain.pid';
const DRAIN_NEWFILE = '/var/run/zt-gateway-drain.new';
const DRAIN_OLDFILE = '/var/run/zt-gateway-drain.old';
function load_gateways() {
const cursor = uci.cursor();
const result = [];
const sections = uci.sections(cursor, 'zt-gateway', 'gateway');
for (let i = 0; i < length(sections); i++) {
const s = sections[i];
push(result, {
region: s.region ?? '',
label: s.label ?? s.region ?? '',
ip: s.ip ?? '',
default: s.default === '1',
health_check: s.health_check ?? 'ping'
});
}
return result;
}
function gateway_by_region(region) {
const gws = load_gateways();
for (let i = 0; i < length(gws); i++) {
if (gws[i].region === region) {
return gws[i];
}
}
return null;
}
function read_active_region() {
const cursor = uci.cursor();
return cursor.get('zt-gateway', 'global', 'active_gateway') ?? '';
}
function read_global_option(option, default_value) {
const cursor = uci.cursor();
return cursor.get('zt-gateway', 'global', option) ?? default_value;
}
function drain_active() {
const st = fs.stat(DRAIN_PIDFILE);
return st?.type === 'file';
}
function read_drain_pid() {
const data = fs.readfile(DRAIN_PIDFILE);
if (!data) {
return null;
}
const s = trim(data);
return length(s) > 0 ? +s : null;
}
function read_drain_state() {
let new_ip = '';
let old_ip = '';
const n = fs.readfile(DRAIN_NEWFILE);
if (n) {
new_ip = trim(n);
}
const o = fs.readfile(DRAIN_OLDFILE);
if (o) {
old_ip = trim(o);
}
return { new_ip, old_ip };
}
function run_switch_script_capture(ip, mode) {
const mode_lc = mode ?? 'force';
let cmd = `${SWITCH_SCRIPT} ${shell_quote(ip)} ${shell_quote(mode_lc)}`;
if (mode_lc === 'graceful') {
const to = +read_global_option('drain_timeout', '600');
cmd += ` ${to > 0 ? to : 600}`;
}
const stderr_pipe = '/tmp/zt-gw-stderr';
const code = system(`${cmd} 2>${stderr_pipe}`);
const err = fs.readfile(stderr_pipe) ?? '';
fs.unlink(stderr_pipe);
return { code: code ?? 0, stderr: trim(err) };
}
function run_switch_script_force_capture(ip) {
const stderr_pipe = '/tmp/zt-gw-stderr';
const code = system(`${SWITCH_SCRIPT} ${shell_quote(ip)} force 2>${stderr_pipe}`);
const err = fs.readfile(stderr_pipe) ?? '';
fs.unlink(stderr_pipe);
return { code: code ?? 0, stderr: trim(err) };
}
function system_output(cmd) {
const tmp = `/tmp/zt-gw-out-${getpid()}`;
system(`${cmd} >${tmp} 2>/dev/null`);
const out = fs.readfile(tmp) ?? '';
fs.unlink(tmp);
return out;
}
function shell_quote(s) {
if (!s) {
return "''";
}
if (s ~ /^[A-Za-z0-9_./:@=-]+$/) {
return s;
}
return `'${replace(s, "'", "'\\''")}'`;
}
function rpc_status(req) {
const gws = load_gateways();
const active_region = read_active_region();
let active_ip = '';
const active_gw = gateway_by_region(active_region);
if (active_gw) {
active_ip = active_gw.ip;
}
const drain_state = read_drain_state();
let remaining = 0;
if (drain_active()) {
const out = system_output(`conntrack -L -m 0x100 2>/dev/null | wc -l`);
remaining = +trim(out) || 0;
}
const cursor = uci.cursor();
const settings = {
switch_mode: cursor.get('zt-gateway', 'global', 'switch_mode') ?? 'force',
drain_timeout: +(cursor.get('zt-gateway', 'global', 'drain_timeout') ?? 600),
health_interval: +(cursor.get('zt-gateway', 'global', 'health_interval') ?? 60),
auto_failback: cursor.get('zt-gateway', 'global', 'auto_failback') === '1'
};
ubus.reply(req, {
active_gateway: active_region,
active_ip: active_ip,
settings: settings,
gateways: gws,
drain: {
active: drain_active(),
new_ip: drain_state.new_ip,
old_ip: drain_state.old_ip,
remaining_connections: remaining
}
});
}
function rpc_switch(req, msg) {
const region = msg?.region;
const mode = msg?.mode ?? read_global_option('switch_mode', 'force');
if (!region || (mode !== 'force' && mode !== 'graceful')) {
ubus.reply(req, {
success: false,
message: 'Missing or invalid argument: region required, mode must be force|graceful'
});
return;
}
if (drain_active() && mode === 'graceful') {
ubus.reply(req, {
success: false,
message: 'A graceful drain is already in progress. Cancel it first.'
});
return;
}
const gw = gateway_by_region(region);
if (!gw) {
ubus.reply(req, { success: false, message: `Unknown gateway region: ${region}` });
return;
}
const current_region = read_active_region();
if (current_region === region && !drain_active()) {
ubus.reply(req, { success: true, message: `Already on ${region}.` });
return;
}
const result = run_switch_script_capture(gw.ip, mode);
if (result.code !== 0) {
ubus.reply(req, {
success: false,
message: result.stderr || `Switch script failed with exit code ${result.code}`
});
return;
}
const cursor = uci.cursor();
cursor.set('zt-gateway', 'global', 'active_gateway', region);
cursor.commit('zt-gateway');
ubus.reply(req, {
success: true,
message: `Switched to ${region} (${gw.ip}) via ${mode}`,
active_gateway: region,
mode: mode,
drain_active: (mode === 'graceful')
});
}
function rpc_health(req, msg) {
const region = msg?.region;
if (!region) {
ubus.reply(req, { reachable: false, message: 'Missing region' });
return;
}
const gw = gateway_by_region(region);
if (!gw) {
ubus.reply(req, { reachable: false, message: `Unknown region ${region}` });
return;
}
const out = system_output(
`ping -c 2 -W 3 -I br-zt ${shell_quote(gw.ip)} 2>/dev/null; echo "EXIT:$?"`
);
const exit_match = match(out, /EXIT:(\d+)/);
const exit_code = exit_match ? +exit_match[1] : 1;
let latency_ms = null;
if (exit_code === 0) {
const stats = match(out, /rtt min\/avg\/max[^=]*=\s*[\d.]+\/([\d.]+)\//);
if (stats) {
latency_ms = +stats[1];
}
}
ubus.reply(req, {
region: region,
ip: gw.ip,
reachable: exit_code === 0,
latency_ms: latency_ms
});
}
function rpc_drain_status(req) {
const active = drain_active();
const state = read_drain_state();
let remaining = 0;
if (active) {
const out = system_output(`conntrack -L -m 0x100 2>/dev/null | wc -l`);
remaining = +trim(out) || 0;
}
ubus.reply(req, {
active: active,
new_ip: state.new_ip,
old_ip: state.old_ip,
remaining_connections: remaining
});
}
function rpc_cancel_drain(req) {
if (!drain_active()) {
ubus.reply(req, { success: false, message: 'No drain in progress' });
return;
}
const state = read_drain_state();
const pid = read_drain_pid();
if (pid) {
system(`kill -TERM ${pid} 2>/dev/null`);
}
if (!state.new_ip) {
fs.unlink(DRAIN_PIDFILE);
ubus.reply(req, {
success: true,
message: 'Drain cancelled (new gateway unknown; route left as-is).'
});
return;
}
const result = run_switch_script_force_capture(state.new_ip);
fs.unlink(DRAIN_PIDFILE);
fs.unlink(DRAIN_NEWFILE);
fs.unlink(DRAIN_OLDFILE);
ubus.reply(req, {
success: result.code === 0,
message: result.code === 0
? `Drain cancelled; force-switched to ${state.new_ip}.`
: (result.stderr || `Cancel failed with code ${result.code}`)
});
}
export default {
status: rpc_status,
switch: rpc_switch,
health: rpc_health,
drain_status: rpc_drain_status,
cancel_drain: rpc_cancel_drain
};