infra/luci-app-zt-gateway
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
luci-app-zt-gateway/.omp/plans/zerotier-gateway-switching.md

1258 lines
47 KiB
Markdown
Raw Permalink Normal View History

chore: track .omp/ harness directory
2026-07-12 23:43:36 +05:30
# Plan: luci-app-zt-gateway — ZeroTier Exit Gateway Switcher
## Goal
A LuCI app for OpenWRT that lets users switch which remote ZeroTier node acts as the internet exit gateway for WIBLAN clients (`10.11.13.0/24`). Users select a region/gateway, choose a switch mode, and the app reconfigures routing with zero or minimal disruption.
## Existing Architecture (constraints)
- **Router**: OpenWRT, `192.168.13.1`
- **ZT network**: `e3918db48378cb6a`, interface `ztk4jpk77j`, bridge `br-zt`
- **Current gateway**: `10.11.12.3` (ocirosea641, amsterdam)
- **WIBLAN clients**: `10.11.13.0/24` routed via `ip rule` priority 100 → table 100
- **Table 100**: `default via <gateway> dev br-zt`
- **Host route**: `<gateway> dev br-zt` (ARP fix — gateway IP not directly reachable via bridge)
- **mwan3**: uses table 1; requires `10.11.13.0/24 dev br-zt table 1` for return traffic
- **Hotplug**: `/etc/hotplug.d/net/99-zerotier-bridge` adds `ztk4jpk77j` to `br-zt`, removes conflicting routes, adds table 1 route
- **Conntrack**: `conntrack` package available; `conntrack -D -s 10.11.13.0/24` flushes WIBLAN entries
## Two Switch Modes
### 1. Force Switch (instant, ~1-2s disruption)
All existing connections break immediately. Clients auto-reconnect through the new gateway.
**Steps:**
1. Pre-flight: ping new gateway over ZT, abort if unreachable
2. Atomically replace host route + table 100 default route
3. Flush conntrack for `10.11.13.0/24`
4. Update hotplug script + rc.local with new gateway IP
5. Update UCI network routes for persistence
6. Verify: `ip route get 8.8.8.8 from 10.11.13.1 table 100`
**User impact:** Active downloads/video calls drop and reconnect. Web pages refresh. SSH sessions die.
### 2. Graceful Drain (zero disruption, complex)
Existing connections continue on the old gateway until they naturally close. New connections go through the new gateway immediately. Uses fwmark + dual routing table.
**Steps:**
1. Pre-flight: ping new gateway over ZT, abort if unreachable
2. Create drain table (101) with `default via <OLD_GATEWAY> dev br-zt`
3. Add iptables mangle rules to mark all ESTABLISHED connections from WIBLAN with fwmark `0x100`
4. Add `ip rule add fwmark 0x100 table 101 priority 99` (before the existing priority-100 src rule)
5. Update table 100 default to new gateway
6. Update host route to new gateway
7. Start drain monitor: count conntrack entries with mark; when zero → clean up mangle rules, drain table, fwmark rule
8. Update hotplug script + rc.local + UCI network routes for persistence
9. If drain doesn't complete within configurable timeout (default 10 min), auto-fallback to force switch
**Key insight:** fwmark 0x100 at priority 99 matches BEFORE the src `10.11.13.0/24` → table 100 rule. So marked (existing) connections go to table 101 (old gateway), while unmarked (new) connections go to table 100 (new gateway).
**User impact:** Zero disruption. Active connections keep working. New connections get new gateway latency/exit point.
**Conntrack marking detail:**
```sh
# Restore mark from conntrack for returning packets
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
# Mark ESTABLISHED packets that don't have a mark yet
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark 0x100
# Save packet mark back to conntrack
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -j CONNMARK --save-mark
```
**Drain monitor:** Periodic check via `conntrack -L -m 0x100 2>/dev/null | wc -l`. When count is 0, clean up.
## File Structure
```
luci-app-zt-gateway/
├── Makefile
├── htdocs/
│ └── luci-static/
│ └── resources/
│ └── view/
│ └── zt-gateway/
│ └── overview.js # Main UI view
├── root/
│ ├── etc/
│ │ └── config/
│ │ └── zt-gateway # UCI config: gateway registry + active state
│ ├── usr/
│ │ ├── share/
│ │ │ ├── acl.d/
│ │ │ │ └── luci-app-zt-gateway.json # ACL permissions
│ │ │ ├── luci/
│ │ │ │ └── menu.d/
│ │ │ │ └── luci-app-zt-gateway.json # Menu entry
│ │ │ └── rpcd/
│ │ │ └── ucode/
│ │ │ └── zt-gateway.uc # rpcd backend: switch, status, health, drain
│ │ └── sbin/
│ │ └── zt-gateway-switch # Shell script: actual switch logic (called by rpcd)
```
## UCI Config Schema (`/etc/config/zt-gateway`)
```
config global 'global'
option active_gateway '' # Region key of active gateway
option switch_mode 'force' # 'force' or 'graceful'
option drain_timeout '600' # Graceful drain timeout in seconds (default 10 min)
option health_interval '60' # Health check interval in seconds
option auto_failback '0' # Auto-failback to default on gateway down
config gateway
option region 'amsterdam'
option label 'Amsterdam (ocirosea641)'
option ip '10.11.12.3'
option default '1' # Default gateway on boot
option health_check 'ping' # 'ping' | 'tcp' | 'none'
config gateway
option region 'tirunelveli'
option label 'Tirunelveli (rpi1000)'
option ip '10.11.12.5'
option default '0'
option health_check 'ping'
config gateway
option region 'bangalore'
option label 'Bangalore (sensecap-m4)'
option ip '10.11.12.4'
option default '0'
option health_check 'ping' # Not yet joined to ZT network; assign IP after joining
```
## rpcd Backend (`/usr/share/rpcd/ucode/zt-gateway.uc`)
Exposes these ubus methods:
| Method | Args | Returns | Description |
|---|---|---|---|
| `status` | — | `{ active_gateway, gateways: [...], drain_status? }` | Current state + all configured gateways |
| `switch` | `{ region, mode }` | `{ success, message }` | Execute gateway switch |
| `health` | `{ region }` | `{ reachable, latency_ms }` | Ping a gateway via ZT |
| `drain_status` | — | `{ active, remaining_connections }` | Graceful drain progress |
| `cancel_drain` | — | `{ success }` | Cancel graceful drain, force switch to new gateway |
## Shell Script (`/usr/sbin/zt-gateway-switch`)
The rpcd backend calls this script for the actual system changes. Separating it from ucode makes it testable from SSH.
```
Usage: zt-gateway-switch <new_ip> <mode> [drain_timeout]
Modes:
force - Instant switch + conntrack flush
graceful - fwmark drain with timeout fallback to force
```
**Force switch logic:**
1. Validate new gateway IP reachable: `ping -c 2 -W 3 <new_ip>`
2. `ip route replace <new_ip> dev br-zt` (host route)
3. `ip route replace default via <new_ip> dev br-zt table 100`
4. `conntrack -D -s 10.11.13.0/24 2>/dev/null`
5. Update `/etc/hotplug.d/net/99-zerotier-bridge` with new gateway IP
6. Update `/etc/rc.local` with new gateway IP
7. Update UCI network routes: `zt_gateway_host`, `zt_gateway_default`
8. `uci commit network`
9. Verify new route
**Graceful switch logic:**
1. Validate new gateway IP reachable
2. Read current gateway from UCI
3. Create table 101: `ip route add default via <OLD_IP> dev br-zt table 101`
4. Add mangle rules (fwmark 0x100 for ESTABLISHED)
5. Add `ip rule add fwmark 0x100 table 101 priority 99`
6. Update table 100 default + host route to new gateway
7. Start drain monitor loop:
- Every 5s: `conntrack -L -m 0x100 2>/dev/null | wc -l`
- If count == 0: clean up mangle rules, fwmark rule, table 101 → done
- If timeout exceeded: force flush + clean up → done
8. Update hotplug, rc.local, UCI
**Cleanup function** (called after drain completes or on cancel):
1. `iptables -t mangle -D <each rule>`
2. `ip rule del fwmark 0x100 table 101 priority 99`
3. `ip route del default dev br-zt table 101`
## LuCI Frontend (`overview.js`)
### Layout
```
┌─────────────────────────────────────────────────┐
│ ZeroTier Exit Gateway │
├─────────────────────────────────────────────────┤
│ │
│ Active Gateway: amsterdam (10.11.12.3) ● UP │
│ Switch Mode: [Force v] │
│ Drain Timeout: 600s (shown if graceful) │
│ │
│ ┌─────────────────────────────────────────────┐│
│ │ Region │ Label │ IP ││
│ ├─────────────┼──────────────────┼─────────────┤│
│ │ ● amsterdam │ Amsterdam │ 10.11.12.3 ││
│ │ ○ tirunelveli│ Tirunelveli │ 10.11.12.5 ││
│ │ ○ bangalore │ Bangalore │ 10.11.12.4 ││
│ └─────────────┴──────────────────┴─────────────┘│
│ │
│ [Switch to Selected] │
│ │
│ ── Status ── │
│ Drain progress: 14 connections remaining │
│ [Cancel Drain → Force Switch] │
│ │
└─────────────────────────────────────────────────┘
```
### JavaScript architecture
- Form.Map tied to `zt-gateway` UCI config for settings section (switch mode, drain timeout)
- Radio-button list for gateway selection (reads `config gateway` sections)
- Custom button handler for "Switch to Selected" — calls `zt-gateway.switch` RPC
- Polling for drain status when graceful switch is in progress
- Health status indicator (green/red dot) per gateway, refreshed every `health_interval`
### Key imports
```js
'use strict';
import { view } from 'luci.view';
import { form } from 'luci.form';
import { rpc } from 'luci.rpc';
import { ui } from 'luci.ui';
import { dom } from 'luci.dom';
import { Poll } from 'luci.poll';
```
## Persistence Model
The gateway IP appears in **five places** on the router. The switch script updates them all atomically:
| Location | What | How updated |
|---|---|---|
| Kernel: host route | `<gw_ip> dev br-zt` | `ip route replace` |
| Kernel: table 100 default | `default via <gw_ip> dev br-zt table 100` | `ip route replace` |
| UCI: `/etc/config/network` | `zt_gateway_host` + `zt_gateway_default` routes | `uci set` + `uci commit` |
| Hotplug: `99-zerotier-bridge` | host route + table 100 route + table 1 return | `sed` replace |
| Boot: `/etc/rc.local` | all three routes | `sed` replace |
The hotplug and rc.local currently hardcode `10.11.12.3`. The script replaces these with the new gateway IP. A cleaner future approach would be to have the hotplug/rc.local read from UCI, but that's a refactoring step.
## Health Check
Default method: `ping -c 2 -W 3 <gateway_ip>` over the ZT network.
- Runs every `health_interval` seconds via cron or the LuCI polling mechanism
- If active gateway becomes unreachable AND `auto_failback='1'`: switch to the `default='1'` gateway
- Health status displayed in UI with latency
## Rollback / Safety
- Pre-flight: refuse to switch if new gateway is unreachable
- Graceful drain timeout: if drain doesn't complete, auto-fallback to force
- If switch script fails partway: the host route + table 100 are updated atomically; worst case the old conntrack entries persist and time out naturally
- Manual recovery: `uci set zt-gateway.global.active_gateway='<region>'` + run `/usr/sbin/zt-gateway-switch <ip> force`
## Files to Create
| File | Purpose |
|---|---|
| `luci-app-zt-gateway/Makefile` | OpenWRT build system |
| `luci-app-zt-gateway/root/etc/config/zt-gateway` | UCI config skeleton |
| `luci-app-zt-gateway/root/usr/share/acl.d/luci-app-zt-gateway.json` | RPC + UCI ACL |
| `luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | Menu entry |
| `luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc` | rpcd backend |
| `luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch` | Switch script |
| `luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js` | Frontend UI |
## Verification
1. Install package on router
2. Configure 2+ gateways in `/etc/config/zt-gateway`
3. **Force switch test**: select new region → Force → verify `ip route show table 100` shows new gateway, `curl ifconfig.me` from WIBLAN client shows new exit IP
4. **Graceful switch test**: select new region → Graceful → verify existing SSH on WIBLAN client stays alive, new connections go through new gateway, drain counter decrements
5. **Fail test**: switch to unreachable gateway → verify pre-flight blocks the switch with error message
6. **Drain timeout test**: graceful switch with 30s timeout → verify it falls back to force after timeout
7. **Reboot persistence**: switch gateway → reboot router → verify table 100 still has the new gateway
## Future: ZeroTier Central API Integration
Skipping for now. When added, the switch script should:
1. Call `DELETE /api/network/{networkId}/route/{routeId}` for the old `0.0.0.0/0` managed route
2. Call `POST /api/network/{networkId}/route` with `{ "target": "0.0.0.0/0", "via": "<new_gateway_ip>" }`
3. Toggle "Allow Default Route Override" on new gateway node, disable on old
Requires storing a ZeroTier Central API token in UCI:
```
config global 'global'
option zt_central_token ''
option zt_network_id 'e3918db48378cb6a'
```
This is only needed so other ZeroTier peers also get the correct default route — WIBLAN clients are already handled by the local `ip rule`.
## Future: Gateway Auto-Provisioning
Currently, each gateway node must be manually set up with:
- ZeroTier membership + static IP
- IP forwarding + NAT
- iptables FORWARD rules
A future enhancement could SSH into candidate nodes from the controller and set up NAT automatically, or use an agent on each gateway that phones home. Out of scope for v1.
---
## Building and Installing luci-app-zt-gateway
This section covers how to build the `luci-app-zt-gateway` package from source into an `.ipk` using the OpenWRT SDK, then install it on the target router.
### Prerequisites
- A **Linux** development machine (Ubuntu 22.04+ / Debian 12+ recommended)
- **~2 GB** free disk space for the SDK
- Basic build tools: `gcc`, `make`, `perl`, `python3`, `zstd`
- SSH/SCP access to the target router (`192.168.13.1`)
- The target router's **OpenWRT version** and **architecture** (determine these before downloading the SDK)
#### Determine router architecture
SSH into the router and run:
```sh
cat /etc/openwrt_release # shows OPENWRT_RELEASE, OPENWRT_ARCH, OPENWRT_BOARD
opkg print-architecture # lists available architectures
```
The `OPENWRT_ARCH` line tells you the target (e.g. `mipsel_24kc`, `aarch64_cortex-a53`, `x86_64`).
### Step 1: Download the OpenWRT SDK
Download the SDK matching your router's **exact OpenWRT version and architecture** from the OpenWRT downloads server:
- **OpenWRT 24.10**: https://downloads.openwrt.org/releases/24.10.7/targets/
- **OpenWRT 23.05**: https://downloads.openwrt.org/releases/23.05.5/targets/
Navigate into the appropriate target subdirectory (e.g. `x86/64/`, `ramips/mt7621/`, `mediatek/filogic/`) and download the `openwrt-sdk-*.tar.zst` (or `.tar.xz`) file.
```sh
# Example for x86_64 / OpenWRT 24.10
wget https://downloads.openwrt.org/releases/24.10.7/targets/x86/64/openwrt-sdk-24.10.7-x86-64_gcc-13.3.0_musl.Linux-x86_64.tar.zst
# Extract
tar --zstd -xf openwrt-sdk-24.10.7-*.tar.zst
cd openwrt-sdk-24.10.7-*
```
### Step 2: Prepare the SDK
```sh
# Update and install standard feed definitions
./scripts/feeds update -a
./scripts/feeds install -a
```
### Step 3: Add the package source
Place the `luci-app-zt-gateway` directory into the SDK's package tree. There are two approaches:
#### Option A: Direct placement (simplest)
```sh
# Copy or symlink the package directory into the SDK
# If your source is at ~/projects/luci-app-zt-gateway/:
ln -s ~/projects/luci-app-zt-gateway package/luci-app-zt-gateway
```
#### Option B: Custom feed (recommended for multiple packages)
```sh
# 1. Create a feed directory
mkdir -p ~/my-openwrt-feed
ln -s ~/projects/luci-app-zt-gateway ~/my-openwrt-feed/luci-app-zt-gateway
# 2. Register the feed in the SDK
echo 'src-link customfeed /home/YOURUSER/my-openwrt-feed' >> feeds.conf.default
# 3. Update and install the custom feed
./scripts/feeds update customfeed
./scripts/feeds install -p customfeed luci-app-zt-gateway
```
### Step 4: The Makefile
The `luci-app-zt-gateway/Makefile` uses the LuCI build system (`luci.mk`) which automates directory mapping, installation, and packaging. The Makefile must be in the package root alongside the `htdocs/` and `root/` directories.
```makefile
# luci-app-zt-gateway/Makefile
include $(TOPDIR)/rules.mk
PKG_NAME:=luci-app-zt-gateway
LUCI_TITLE:=LuCI support for ZeroTier Exit Gateway Switching
LUCI_DEPENDS:=+luci-base +ucode +conntrack
LUCI_PKGARCH:=all
PKG_VERSION:=1.0.0
PKG_RELEASE:=1
PKG_LICENSE:=Apache-2.0
PKG_MAINTAINER:=Your Name <you@example.com>
include $(TOPDIR)/feeds/luci/luci.mk
# call BuildPackage - OpenWrt buildroot signature
```
Key points:
- `luci.mk` (not `package.mk`) is used for LuCI applications. It scans `htdocs/` and `root/` and automatically creates install rules that map files to their target paths:
- `htdocs/luci-static/resources/view/` → `/www/luci-static/resources/view/`
- `root/` → mirrors directly onto the target filesystem root (`/`)
- `LUCI_PKGARCH:=all` because LuCI apps are architecture-independent (JS, ucode, shell only).
- `include $(TOPDIR)/feeds/luci/luci.mk` uses the absolute path variant. If building inside the main `openwrt/luci` feed, use `include ../../luci.mk` instead.
- The trailing comment `# call BuildPackage - OpenWrt buildroot signature` is **required** by the OpenWRT feed scanner to discover this package.
- **Dependencies**: `+luci-base` is mandatory. `+ucode` is needed for the rpcd backend. `+conntrack` is needed because the switch script uses `conntrack` CLI.
#### Directory-to-path mapping (luci.mk automatics)
When `luci.mk` processes this package, it automatically installs:
| Source path in package | Target path on router |
|---|---|
| `htdocs/luci-static/resources/view/zt-gateway/overview.js` | `/www/luci-static/resources/view/zt-gateway/overview.js` |
| `root/etc/config/zt-gateway` | `/etc/config/zt-gateway` |
| `root/usr/share/rpcd/ucode/zt-gateway.uc` | `/usr/share/rpcd/ucode/zt-gateway.uc` |
| `root/usr/share/acl.d/luci-app-zt-gateway.json` | `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` |
| `root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | `/usr/share/luci/menu.d/luci-app-zt-gateway.json` |
| `root/usr/sbin/zt-gateway-switch` | `/usr/sbin/zt-gateway-switch` |
### Step 5: Configure and build
```sh
# Select the package in menuconfig
make menuconfig
# Navigate: LuCI → 3. Applications → luci-app-zt-gateway
# Press 'M' to build as module (do NOT press 'Y' — we want an .ipk, not a firmware image)
# Save and exit
# Build the package with verbose output
make package/luci-app-zt-gateway/compile V=s
```
If `make menuconfig` doesn't show the package, ensure the feeds are updated:
```sh
./scripts/feeds update -a
./scripts/feeds install luci-app-zt-gateway
```
### Step 6: Locate the built .ipk
```sh
find bin/ -name 'luci-app-zt-gateway*.ipk' -type f
```
Typical output path: `bin/packages/<architecture>/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk`
Because `LUCI_PKGARCH:=all`, the `.ipk` filename ends with `_all.ipk` — it's architecture-independent and works on any OpenWRT target.
### Step 7: Install on the router
#### Transfer the package
```sh
scp bin/packages/*/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk root@192.168.13.1:/tmp/
```
#### Install via opkg (OpenWRT ≤ 24.10)
```sh
ssh root@192.168.13.1
opkg update
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```
If you get dependency errors (e.g. missing `luci-base`), install them first:
```sh
opkg install luci-base ucode conntrack
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```
#### Install via apk (OpenWRT 25.12+)
Starting with OpenWRT 25.12, the package manager is `apk` instead of `opkg`:
```sh
ssh root@192.168.13.1
apk add --allow-untrusted /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```
#### Post-install: refresh LuCI cache
```sh
rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
```
Then navigate to the LuCI web UI — the "ZeroTier Exit Gateway" page should appear under **Services**.
### Step 8: Configure gateways
After installation, edit the UCI config to add your gateway nodes:
```sh
vi /etc/config/zt-gateway
```
Or use `uci` commands:
```sh
uci set zt-gateway.global.active_gateway='amsterdam'
uci set zt-gateway.global.switch_mode='force'
uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='amsterdam'
uci set zt-gateway.@gateway[-1].label='Amsterdam (ocirosea641)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.3'
uci set zt-gateway.@gateway[-1].default='1'
uci set zt-gateway.@gateway[-1].health_check='ping'
uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='tirunelveli'
uci set zt-gateway.@gateway[-1].label='Tirunelveli (rpi1000)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.5'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'
uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='bangalore'
uci set zt-gateway.@gateway[-1].label='Bangalore (sensecap-m4)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.4'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'
uci commit zt-gateway
```
### Step 9: Verify the rpcd backend is registered
After install, verify the ubus backend is available:
```sh
# Restart rpcd to pick up the new backend
/etc/init.d/rpcd restart
# List available methods
ubus list zt-gateway.*
# Expected output:
# zt-gateway.status
# zt-gateway.switch
# zt-gateway.health
# zt-gateway.drain_status
# zt-gateway.cancel_drain
# Test the status method
ubus call zt-gateway status
```
If `ubus list` doesn't show `zt-gateway.*`, check:
1. The ucode script is executable and at the correct path: `ls -la /usr/share/rpcd/ucode/zt-gateway.uc`
2. The ACL file exists: `cat /usr/share/rpcd/acl.d/luci-app-zt-gateway.json`
3. rpcd log output: `logread -e rpcd`
### Quick-build cheat sheet (no SDK, manual .ipk)
If you don't want to bother with the full SDK and just want to produce an `.ipk` manually, you can use the `opkg-utils` tools on any Linux machine:
```sh
# Install opkg-utils
sudo apt install opkg-utils # Debian/Ubuntu
# Create the package staging directory
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin
# Copy source files into staging (adjust source paths as needed)
cp htdocs/luci-static/resources/view/zt-gateway/overview.js \
/tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway/
cp root/etc/config/zt-gateway /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config/
cp root/usr/share/rpcd/ucode/zt-gateway.uc /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode/
cp root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d/
cp root/usr/share/luci/menu.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d/
cp root/usr/sbin/zt-gateway-switch /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/
chmod +x /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/zt-gateway-switch
# Write the control file
cat > /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL/control << 'EOF'
Package: luci-app-zt-gateway
Version: 1.0.0-1
Depends: libc, luci-base, ucode, conntrack
Source: luci-app-zt-gateway
Section: luci
Architecture: all
Installed-Size: 20480
Description: LuCI support for ZeroTier Exit Gateway Switching
EOF
# Build the .ipk
cd /tmp
opkg-build -O -o root -g root luci-app-zt-gateway_1.0.0-1_all
# Result: /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```
This manual approach skips the SDK entirely. It's suitable for quick iteration during development. The tradeoff: no automatic dependency resolution, no translation compilation, no JS minification.
### Development workflow: iterate fast without rebuilding
During development, you can skip the build step entirely by editing files directly on the router:
```sh
# Edit the JS view directly
scp htdocs/luci-static/resources/view/zt-gateway/overview.js \
root@192.168.13.1:/www/luci-static/resources/view/zt-gateway/overview.js
# Edit the ucode backend
scp root/usr/share/rpcd/ucode/zt-gateway.uc \
root@192.168.13.1:/usr/share/rpcd/ucode/zt-gateway.uc
/etc/init.d/rpcd restart
# Edit the switch script
scp root/usr/sbin/zt-gateway-switch \
root@192.168.13.1:/usr/sbin/zt-gateway-switch
# Edit the UCI config
scp root/etc/config/zt-gateway \
root@192.168.13.1:/etc/config/zt-gateway
# Clear LuCI browser cache after JS changes
ssh root@192.168.13.1 'rm -rf /tmp/luci-indexcache /tmp/luci-modulecache'
```
Build the `.ipk` only when you need a clean install or are sharing the package.
### Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| `make menuconfig` doesn't show the package | Feed not registered | Run `./scripts/feeds update -a && ./scripts/feeds install -a` |
| `opkg install` fails with "cannot install package" | Missing dependencies | Run `opkg update` then install dependencies first |
| LuCI page shows 404 | JS view not found at expected path | Verify `/www/luci-static/resources/view/zt-gateway/overview.js` exists on router |
| ubus methods missing | rpcd didn't pick up the ucode backend | `/etc/init.d/rpcd restart`; check `logread -e rpcd` for errors |
| ACL denied when calling ubus from LuCI | ACL JSON file missing or wrong format | Verify `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` and restart rpcd |
| `luci.mk` not found during build | Building outside the LuCI feed | Use `include $(TOPDIR)/feeds/luci/luci.mk` (absolute) instead of `include ../../luci.mk` (relative) |
| JS view loads but is blank | Import path errors in JS | Open browser dev console; LuCI JS modules use `luci.view`, `luci.form`, etc. |
## Local Testing with Docker (macvlan)
This section covers how to test `luci-app-zt-gateway` locally using the official `openwrt/rootfs` Docker image with macvlan networking. This gives you a full OpenWRT environment with real `ip rule`, `iptables`, `conntrack`, and rpcd — no physical router required.
### Test architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Docker host (your dev machine) │
│ │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ openwrt-router │ │ wiblan-client │ │
│ │ (macvlan 10.11.13.1)│ │ (macvlan 10.11.13.10)│ │
│ │ │ │ │ │
│ │ - LuCI on :80 │ │ - curl / ping │ │
│ │ - rpcd + ucode │ │ - traffic exits via │ │
│ │ - zt-gateway-switch │ │ router's table 100 │ │
│ │ - br-zt + ip rules │ │ │ │
│ └──────────┬───────────┘ └───────────┬───────────┘ │
│ │ │ │
│ └──────────┬─────────────────┘ │
│ │ │
│ zt-gateway-macvlan │
│ 10.11.13.0/24 subnet │
│ (macvlan bridge mode) │
│ │
│ ┌──────────────────────┐ │
│ │ zt-gw-amsterdam │ ← Simulates a remote ZeroTier exit │
│ │ (macvlan 10.11.12.3) │ node with NAT + ip_forward │
│ │ │ │
│ │ - ip_forward=1 │ │
│ │ - iptables MASQ │ │
│ └──────────────────────┘ │
│ │
│ zt-exit-macvlan │
│ 10.11.12.0/24 subnet │
└──────────────────────────────────────────────────────────────────┘
```
The key insight: we use **two macvlan networks** to simulate the two network segments the real router has:
1. **zt-gateway-macvlan** (`10.11.13.0/24`) — represents the WIBLAN subnet. The router sits at `.1`, client at `.10`.
2. **zt-exit-macvlan** (`10.11.12.0/24`) — represents the ZeroTier network where exit gateway nodes live.
### Step 1: Create the Dockerfile
Create a Dockerfile that builds a pre-configured OpenWRT image with all dependencies and the zt-gateway package:
```dockerfile
# Dockerfile.openwrt-dev
FROM openwrt/rootfs:x86-64-24.10.7
# Install required packages
RUN opkg update && opkg install \
luci \
luci-base \
ucode \
conntrack \
ip-bridge \
iptables \
kmod-ipt-conntrack \
kmod-ipt-connmark \
kmod-ipt-mark \
kmod-ipt-extra \
&& rm -rf /var/opkg-lists/*
# Enable uhttpd
RUN /etc/init.d/uhttpd enable
RUN /etc/init.d/uhttpd start
# Enable rpcd
RUN /etc/init.d/rpcd enable
RUN /etc/init.d/rpcd start
# Set root password (required for LuCI login)
RUN echo "root:root" | chpasswd
EXPOSE 80
CMD ["/sbin/init"]
```
Build it:
```sh
docker build -t openwrt-zt-gateway-dev -f Dockerfile.openwrt-dev .
```
**Note:** The exact `kmod-*` packages available depend on the OpenWRT version. If `opkg install` fails on kernel modules, skip them — the Docker container's kernel is the host kernel, so the modules are already loaded. What matters is that `iptables -t mangle` and `conntrack` work. Verify with:
```sh
# Run a quick check after building
docker run --rm --privileged openwrt-zt-gateway-dev sh -c \
"iptables -t mangle -L && echo 'mangle OK' && conntrack -L 2>/dev/null && echo 'conntrack OK'"
```
### Step 2: Create the Docker Compose file
```yaml
# docker-compose.yml
version: "3.8"
networks:
# Simulates the WIBLAN client network (10.11.13.0/24)
zt-gateway-lan:
driver: macvlan
driver_opts:
mode: bridge
ipam:
config:
- subnet: "10.11.13.0/24"
gateway: "10.11.13.1"
ip_range: "10.11.13.100/28"
# Simulates the ZeroTier exit node network (10.11.12.0/24)
zt-exit-net:
driver: macvlan
driver_opts:
mode: bridge
ipam:
config:
- subnet: "10.11.12.0/24"
ip_range: "10.11.12.100/28"
services:
# The OpenWRT router running luci-app-zt-gateway
openwrt-router:
image: openwrt-zt-gateway-dev
container_name: openwrt-router
privileged: true
networks:
zt-gateway-lan:
ipv4_address: "10.11.13.1"
zt-exit-net:
ipv4_address: "10.11.12.1"
ports:
- "8080:80"
volumes:
# Hot-mount source files for live iteration (key advantage)
- ./luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js
- ./luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc
- ./luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch:/usr/sbin/zt-gateway-switch
- ./luci-app-zt-gateway/root/etc/config/zt-gateway:/etc/config/zt-gateway
- ./luci-app-zt-gateway/root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json
- ./luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json
cap_add:
- NET_ADMIN
- SYS_ADMIN
sysctls:
- net.ipv4.ip_forward=1
- net.ipv4.conf.all.send_redirects=0
command: ["/sbin/init"]
# A WIBLAN client to test routing through the gateway
wiblan-client:
image: alpine:latest
container_name: wiblan-client
networks:
zt-gateway-lan:
ipv4_address: "10.11.13.10"
cap_add:
- NET_ADMIN
command: ["sleep", "infinity"]
# Simulates the amsterdam ZeroTier exit gateway (10.11.12.3)
zt-gw-amsterdam:
image: alpine:latest
container_name: zt-gw-amsterdam
networks:
zt-exit-net:
ipv4_address: "10.11.12.3"
cap_add:
- NET_ADMIN
sysctls:
- net.ipv4.ip_forward=1
command: ["sleep", "infinity"]
# Simulates the tirunelveli ZeroTier exit gateway (10.11.12.5)
zt-gw-tirunelveli:
image: alpine:latest
container_name: zt-gw-tirunelveli
networks:
zt-exit-net:
ipv4_address: "10.11.12.5"
cap_add:
- NET_ADMIN
sysctls:
- net.ipv4.ip_forward=1
command: ["sleep", "infinity"]
```
### Step 3: Launch and configure the environment
```sh
# Start all containers
docker compose up -d
# Wait a few seconds for init, then configure the simulated exit gateways
# These simulate real ZeroTier exit nodes with NAT
```
#### Configure the amsterdam exit gateway (10.11.12.3)
```sh
docker exec -it zt-gw-amsterdam sh -c '
# Enable forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
# Add a default route back to the router so return traffic works
ip route add 10.11.13.0/24 via 10.11.12.1
# Add NAT so traffic from WIBLAN clients can reach the internet
# (or just other container networks for testing)
apk add --no-cache iptables
iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
# Add a fake "internet" endpoint (just the Docker host gateway)
# In real use, this would be the actual internet
ping -c 1 10.11.12.1 2>/dev/null
'
```
#### Configure the tirunelveli exit gateway (10.11.12.5)
```sh
docker exec -it zt-gw-tirunelveli sh -c '
echo 1 > /proc/sys/net/ipv4/ip_forward
ip route add 10.11.13.0/24 via 10.11.12.1
apk add --no-cache iptables
iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
'
```
#### Configure the OpenWRT router
```sh
docker exec -it openwrt-router sh
```
Inside the router container:
```sh
# 1. Create the br-zt bridge (simulates the ZeroTier bridge)
ip link add name br-zt type bridge
ip link set br-zt up
# Assign the ZT-exit IP to the bridge so the router can reach 10.11.12.0/24
ip addr add 10.11.12.1/24 dev br-zt
# 2. Set up policy routing (same as the real router)
# Host route to the current gateway (ARP fix)
ip route replace 10.11.12.3 dev br-zt
# Default route via the current gateway in table 100
ip route replace default via 10.11.12.3 dev br-zt table 100
# Route WIBLAN subnet via br-zt in mwan3 table (for return traffic)
ip route replace 10.11.13.0/24 dev br-zt table 1
# Source-based routing rule: WIBLAN traffic → table 100
ip rule add from 10.11.13.0/24 table 100 priority 100
# 3. Verify routing
ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt
ip rule show
# Expected: 100: from 10.11.13.0/24 lookup 100
# 4. Test from the router itself
ping -c 2 10.11.12.3 # Should reach amsterdam gateway
```
#### Configure the WIBLAN client
```sh
docker exec -it wiblan-client sh -c '
# Add a route to the router for the ZT subnet
ip route add 10.11.12.0/24 via 10.11.13.1
# Set the OpenWRT router as default gateway
ip route add default via 10.11.13.1
# Test connectivity
ping -c 2 10.11.12.3
'
```
### Step 4: Install and test the luci-app-zt-gateway package
#### Option A: The package is already volume-mounted
If you used the `volumes:` section in `docker-compose.yml`, the files are already in place. Just restart services:
```sh
docker exec -it openwrt-router sh -c '
chmod +x /usr/sbin/zt-gateway-switch
/etc/init.d/rpcd restart
rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
'
```
#### Option B: Install the .ipk
If you built an `.ipk`, copy it into the container:
```sh
docker cp luci-app-zt-gateway_1.0.0-1_all.ipk openwrt-router:/tmp/
docker exec -it openwrt-router opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```
#### Verify the rpcd backend
```sh
docker exec -it openwrt-router ubus list zt-gateway.*
# Expected:
# zt-gateway.status
# zt-gateway.switch
# zt-gateway.health
# zt-gateway.drain_status
# zt-gateway.cancel_drain
docker exec -it openwrt-router ubus call zt-gateway status
```
#### Access the LuCI UI
Open http://localhost:8080 in your browser. Log in with `root` / `root`. Navigate to **Services → ZeroTier Exit Gateway**.
### Step 5: Run the test scenarios
#### Force switch test
```sh
# 1. Verify current gateway is amsterdam (10.11.12.3)
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt
# 2. Trigger a force switch to tirunelveli via the CLI script
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force
# 3. Verify the route changed
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt
# 4. Verify conntrack was flushed
docker exec -it openwrt-router conntrack -L -s 10.11.13.0/24 2>/dev/null
# Expected: empty (all entries flushed)
# 5. Verify from the client side
docker exec -it wiblan-client ping -c 2 10.11.12.5
```
#### Graceful drain test
```sh
# 1. Start a long-running connection from the client
docker exec -d wiblan-client ping 10.11.12.3 # background ping
# 2. Trigger a graceful switch (this will be done from the LuCI UI
# or via ubus call)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 300
# 3. Verify both routing tables exist during drain
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt (new gateway)
docker exec -it openwrt-router ip route show table 101
# Expected: default via 10.11.12.3 dev br-zt (old gateway, drain)
# 4. Verify the fwmark rule exists
docker exec -it openwrt-router ip rule show
# Expected: 99: from all fwmark 0x100 lookup 101
# 5. Verify the mangle rules
docker exec -it openwrt-router iptables -t mangle -L -v
# 6. Watch drain progress
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l
# 7. Wait for drain to complete (or kill the background ping to speed it up)
docker exec -it wiblan-client pkill ping
# 8. After drain completes, verify cleanup
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain table removed)
docker exec -it openwrt-router ip rule show
# Expected: no fwmark 0x100 rule
```
#### Switch to unreachable gateway test
```sh
# 1. Stop the tirunelveli gateway to simulate it being down
docker stop zt-gw-tirunelveli
# 2. Try to switch to it (should fail pre-flight)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force
# Expected: "Gateway unreachable" error, no route changes
# 3. Verify routes unchanged
docker exec -it openwrt-router ip route show table 100
# 4. Restart for subsequent tests
docker start zt-gw-tirunelveli
```
#### Drain timeout test
```sh
# 1. Start a persistent connection
docker exec -d wiblan-client ping 10.11.12.3
# 2. Graceful switch with short timeout (30s)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 30
# 3. Watch the drain monitor — it will hit the 30s timeout
# because the ping keeps conntrack entries alive
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l
# 4. After 30s, should auto-fallback to force switch
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt (force applied)
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain cleaned up by force)
# 5. Clean up
docker exec -it wiblan-client pkill ping
```
### Step 6: Iterate on fixes
The key advantage of the Docker setup is the **volume mounts** in the compose file. When you edit a source file on the host, it's immediately reflected inside the container.
#### Live-editing workflow
```sh
# 1. Edit a source file on the host
vim luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch
# 2. Re-test inside the container (file is already updated via volume mount)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force
# 3. Check results
docker exec -it openwrt-router ip route show table 100
```
For LuCI JS changes, you need to clear the cache:
```sh
# Edit JS on host
vim luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js
# Clear cache inside container
docker exec -it openwrt-router rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
# Refresh browser
```
For ucode backend changes:
```sh
# Edit ucode on host
vim luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc
# Restart rpcd to reload the backend
docker exec -it openwrt-router /etc/init.d/rpcd restart
```
#### Quick-snapshot the exact container state
If you want to save a known-good state before making a risky change:
```sh
# Save container state as a Docker image
docker commit openwrt-router openwrt-zt-gateway-snapshot:good
# ... make changes, break things ...
# Restore from snapshot
docker compose down
docker run -d --name openwrt-router \
--privileged \
-p 8080:80 \
openwrt-zt-gateway-snapshot:good
```
#### Watch logs in real-time
```sh
# OpenWRT system log
docker exec -it openwrt-router logread -f
# rpcd errors specifically
docker exec -it openwrt-router logread -f -e rpcd
# conntrack events in real-time
docker exec -it openwrt-router conntrack -E
```
#### Run the full test suite in one shot
```sh
#!/bin/bash
# test-zt-gateway.sh — Run all test scenarios in Docker
set -e
ROUTER="openwrt-router"
CLIENT="wiblan-client"
GW_AMS="10.11.12.3"
GW_TIR="10.11.12.5"
echo "=== Test 1: Force switch amsterdam → tirunelveli ==="
docker exec $ROUTER zt-gateway-switch $GW_TIR force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo " Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_TIR" && echo " PASS" || echo " FAIL"
echo "=== Test 2: Force switch tirunelveli → amsterdam ==="
docker exec $ROUTER zt-gateway-switch $GW_AMS force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo " Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_AMS" && echo " PASS" || echo " FAIL"
echo "=== Test 3: Switch to unreachable gateway ==="
docker stop zt-gw-tirunelveli
if docker exec $ROUTER zt-gateway-switch $GW_TIR force 2>&1 | grep -qi "unreachable\|fail\|error"; then
echo " PASS (blocked as expected)"
else
echo " FAIL (should have been blocked)"
fi
docker start zt-gw-tirunelveli
sleep 3
echo "=== Test 4: Verify client can reach gateway ==="
docker exec $CLIENT ping -c 2 $GW_AMS && echo " PASS" || echo " FAIL"
echo "=== Test 5: Verify ubus backend ==="
docker exec $ROUTER ubus call zt-gateway status | grep -q "amsterdam" && echo " PASS" || echo " FAIL"
echo "=== All tests complete ==="
```
### Important limitations of the Docker test environment
| Limitation | Impact | Mitigation |
|---|---|---|
| Docker shares the host kernel | `kmod-*` packages from opkg won't load (wrong kernel version) | Use host kernel modules; verify with `lsmod`/`iptables -t mangle -L` |
| `br-zt` is simulated, not a real ZeroTier bridge | No actual ZT membership, no `ztk4jpk77j` interface | Manually create `br-zt` bridge and assign IPs; skip hotplug scripts |
| `conntrack -D -s` may behave differently | Docker's conntrack table includes host traffic too | Use `-s` and `-d` filters; or run `conntrack -F` for a clean slate |
| No real `rc.local` / hotplug execution on boot | Persistence scripts can't be tested end-to-end | Test the switch script logic; verify `sed` replacements on the files separately |
| `iptables -t mangle` requires `NET_ADMIN` | Container must run `--privileged` or `--cap-add=NET_ADMIN` | Already configured in compose file |
| Macvlan host isolation | Host can't ping macvlan containers by default | Use a macvlan shim interface (see below) or just `docker exec` |
| No real ZeroTier Central API | Can't test the future ZT API integration | Mock the API calls or skip this test |
### Fixing the macvlan host-isolation issue
By default, the Docker host **cannot** communicate directly with containers on a macvlan network. If you need to ping/curl the router or client from your host machine (not just via `docker exec`), create a macvlan shim interface:
```sh
# Create a macvlan interface on the host that bridges to the same parent
# Replace eth0 with your host's primary interface
sudo ip link add link eth0 name mv-shim type macvlan mode bridge
sudo ip addr add 10.11.13.254/32 dev mv-shim
sudo ip link set mv-shim up
# Add a route to the WIBLAN subnet via the shim
sudo ip route add 10.11.13.0/24 dev mv-shim
# Now you can reach the containers from host
ping 10.11.13.1 # the router
curl http://10.11.13.1 # LuCI directly (skip the port mapping)
# For the ZT exit subnet
sudo ip link add link eth0 name mv-exit-shim type macvlan mode bridge
sudo ip addr add 10.11.12.254/32 dev mv-exit-shim
sudo ip link set mv-exit-shim up
sudo ip route add 10.11.12.0/24 dev mv-exit-shim
```
Clean up when done:
```sh
sudo ip link del mv-shim
sudo ip link del mv-exit-shim
```
### Teardown
```sh
# Stop and remove all containers
docker compose down
# Remove the shim interfaces (if created)
sudo ip link del mv-shim 2>/dev/null
sudo ip link del mv-exit-shim 2>/dev/null
# Remove the custom images (optional)
docker rmi openwrt-zt-gateway-dev openwrt-zt-gateway-snapshot:good
```
Reference in New Issue Copy Permalink