luci-app-zt-gateway/.omp/plans/zerotier-gateway-switching.md

# Plan: luci-app-zt-gateway — ZeroTier Exit Gateway Switcher

## Goal

A LuCI app for OpenWRT that lets users switch which remote ZeroTier node acts as the internet exit gateway for WIBLAN clients (`10.11.13.0/24`). Users select a region/gateway, choose a switch mode, and the app reconfigures routing with zero or minimal disruption.

## Existing Architecture (constraints)

- **Router**: OpenWRT, `192.168.13.1`
- **ZT network**: `e3918db48378cb6a`, interface `ztk4jpk77j`, bridge `br-zt`
- **Current gateway**: `10.11.12.3` (ocirosea641, amsterdam)
- **WIBLAN clients**: `10.11.13.0/24` routed via `ip rule` priority 100 → table 100
- **Table 100**: `default via <gateway> dev br-zt`
- **Host route**: `<gateway> dev br-zt` (ARP fix — gateway IP not directly reachable via bridge)
- **mwan3**: uses table 1; requires `10.11.13.0/24 dev br-zt table 1` for return traffic
- **Hotplug**: `/etc/hotplug.d/net/99-zerotier-bridge` adds `ztk4jpk77j` to `br-zt`, removes conflicting routes, adds table 1 route
- **Conntrack**: `conntrack` package available; `conntrack -D -s 10.11.13.0/24` flushes WIBLAN entries

## Two Switch Modes

### 1. Force Switch (instant, ~1-2s disruption)

All existing connections break immediately. Clients auto-reconnect through the new gateway.

**Steps:**
1. Pre-flight: ping new gateway over ZT, abort if unreachable
2. Atomically replace host route + table 100 default route
3. Flush conntrack for `10.11.13.0/24`
4. Update hotplug script + rc.local with new gateway IP
5. Update UCI network routes for persistence
6. Verify: `ip route get 8.8.8.8 from 10.11.13.1 table 100`

**User impact:** Active downloads/video calls drop and reconnect. Web pages refresh. SSH sessions die.

### 2. Graceful Drain (zero disruption, complex)

Existing connections continue on the old gateway until they naturally close. New connections go through the new gateway immediately. Uses fwmark + dual routing table.

**Steps:**
1. Pre-flight: ping new gateway over ZT, abort if unreachable
2. Create drain table (101) with `default via <OLD_GATEWAY> dev br-zt`
3. Add iptables mangle rules to mark all ESTABLISHED connections from WIBLAN with fwmark `0x100`
4. Add `ip rule add fwmark 0x100 table 101 priority 99` (before the existing priority-100 src rule)
5. Update table 100 default to new gateway
6. Update host route to new gateway
7. Start drain monitor: count conntrack entries with mark; when zero → clean up mangle rules, drain table, fwmark rule
8. Update hotplug script + rc.local + UCI network routes for persistence
9. If drain doesn't complete within configurable timeout (default 10 min), auto-fallback to force switch

**Key insight:** fwmark 0x100 at priority 99 matches BEFORE the src `10.11.13.0/24` → table 100 rule. So marked (existing) connections go to table 101 (old gateway), while unmarked (new) connections go to table 100 (new gateway).

**User impact:** Zero disruption. Active connections keep working. New connections get new gateway latency/exit point.

**Conntrack marking detail:**
```sh
# Restore mark from conntrack for returning packets
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark ESTABLISHED packets that don't have a mark yet
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -m mark --mark 0 -j MARK --set-mark 0x100

# Save packet mark back to conntrack
iptables -t mangle -A PREROUTING -i br-zt -s 10.11.13.0/24 -j CONNMARK --save-mark
```

**Drain monitor:** Periodic check via `conntrack -L -m 0x100 2>/dev/null | wc -l`. When count is 0, clean up.

## File Structure

```
luci-app-zt-gateway/
├── Makefile
├── htdocs/
│   └── luci-static/
│       └── resources/
│           └── view/
│               └── zt-gateway/
│                   └── overview.js      # Main UI view
├── root/
│   ├── etc/
│   │   └── config/
│   │       └── zt-gateway              # UCI config: gateway registry + active state
│   ├── usr/
│   │   ├── share/
│   │   │   ├── acl.d/
│   │   │   │   └── luci-app-zt-gateway.json  # ACL permissions
│   │   │   ├── luci/
│   │   │   │   └── menu.d/
│   │   │   │       └── luci-app-zt-gateway.json  # Menu entry
│   │   │   └── rpcd/
│   │   │       └── ucode/
│   │   │           └── zt-gateway.uc   # rpcd backend: switch, status, health, drain
│   │   └── sbin/
│   │       └── zt-gateway-switch       # Shell script: actual switch logic (called by rpcd)
```

## UCI Config Schema (`/etc/config/zt-gateway`)

```
config global 'global'
    option active_gateway ''         # Region key of active gateway
    option switch_mode 'force'       # 'force' or 'graceful'
    option drain_timeout '600'       # Graceful drain timeout in seconds (default 10 min)
    option health_interval '60'      # Health check interval in seconds
    option auto_failback '0'         # Auto-failback to default on gateway down

config gateway
    option region 'amsterdam'
    option label 'Amsterdam (ocirosea641)'
    option ip '10.11.12.3'
    option default '1'               # Default gateway on boot
    option health_check 'ping'        # 'ping' | 'tcp' | 'none'

config gateway
    option region 'tirunelveli'
    option label 'Tirunelveli (rpi1000)'
    option ip '10.11.12.5'
    option default '0'
    option health_check 'ping'

config gateway
    option region 'bangalore'
    option label 'Bangalore (sensecap-m4)'
    option ip '10.11.12.4'
    option default '0'
    option health_check 'ping'        # Not yet joined to ZT network; assign IP after joining
```

## rpcd Backend (`/usr/share/rpcd/ucode/zt-gateway.uc`)

Exposes these ubus methods:

| Method | Args | Returns | Description |
|---|---|---|---|
| `status` | — | `{ active_gateway, gateways: [...], drain_status? }` | Current state + all configured gateways |
| `switch` | `{ region, mode }` | `{ success, message }` | Execute gateway switch |
| `health` | `{ region }` | `{ reachable, latency_ms }` | Ping a gateway via ZT |
| `drain_status` | — | `{ active, remaining_connections }` | Graceful drain progress |
| `cancel_drain` | — | `{ success }` | Cancel graceful drain, force switch to new gateway |

## Shell Script (`/usr/sbin/zt-gateway-switch`)

The rpcd backend calls this script for the actual system changes. Separating it from ucode makes it testable from SSH.

```
Usage: zt-gateway-switch <new_ip> <mode> [drain_timeout]

Modes:
  force      - Instant switch + conntrack flush
  graceful   - fwmark drain with timeout fallback to force
```

**Force switch logic:**
1. Validate new gateway IP reachable: `ping -c 2 -W 3 <new_ip>`
2. `ip route replace <new_ip> dev br-zt` (host route)
3. `ip route replace default via <new_ip> dev br-zt table 100`
4. `conntrack -D -s 10.11.13.0/24 2>/dev/null`
5. Update `/etc/hotplug.d/net/99-zerotier-bridge` with new gateway IP
6. Update `/etc/rc.local` with new gateway IP
7. Update UCI network routes: `zt_gateway_host`, `zt_gateway_default`
8. `uci commit network`
9. Verify new route

**Graceful switch logic:**
1. Validate new gateway IP reachable
2. Read current gateway from UCI
3. Create table 101: `ip route add default via <OLD_IP> dev br-zt table 101`
4. Add mangle rules (fwmark 0x100 for ESTABLISHED)
5. Add `ip rule add fwmark 0x100 table 101 priority 99`
6. Update table 100 default + host route to new gateway
7. Start drain monitor loop:
   - Every 5s: `conntrack -L -m 0x100 2>/dev/null | wc -l`
   - If count == 0: clean up mangle rules, fwmark rule, table 101 → done
   - If timeout exceeded: force flush + clean up → done
8. Update hotplug, rc.local, UCI

**Cleanup function** (called after drain completes or on cancel):
1. `iptables -t mangle -D <each rule>`
2. `ip rule del fwmark 0x100 table 101 priority 99`
3. `ip route del default dev br-zt table 101`

## LuCI Frontend (`overview.js`)

### Layout

```
┌─────────────────────────────────────────────────┐
│ ZeroTier Exit Gateway                            │
├─────────────────────────────────────────────────┤
│                                                  │
│  Active Gateway:  amsterdam (10.11.12.3)  ● UP  │
│  Switch Mode:     [Force v]                     │
│  Drain Timeout:   600s  (shown if graceful)     │
│                                                  │
│  ┌─────────────────────────────────────────────┐│
│  │ Region      │ Label            │ IP          ││
│  ├─────────────┼──────────────────┼─────────────┤│
│  │ ● amsterdam │ Amsterdam        │ 10.11.12.3  ││
│  │ ○ tirunelveli│ Tirunelveli     │ 10.11.12.5  ││
│  │ ○ bangalore │ Bangalore        │ 10.11.12.4  ││
│  └─────────────┴──────────────────┴─────────────┘│
│                                                  │
│  [Switch to Selected]                            │
│                                                  │
│  ── Status ──                                    │
│  Drain progress: 14 connections remaining        │
│  [Cancel Drain → Force Switch]                  │
│                                                  │
└─────────────────────────────────────────────────┘
```

### JavaScript architecture

- Form.Map tied to `zt-gateway` UCI config for settings section (switch mode, drain timeout)
- Radio-button list for gateway selection (reads `config gateway` sections)
- Custom button handler for "Switch to Selected" — calls `zt-gateway.switch` RPC
- Polling for drain status when graceful switch is in progress
- Health status indicator (green/red dot) per gateway, refreshed every `health_interval`

### Key imports
```js
'use strict';
import { view } from 'luci.view';
import { form } from 'luci.form';
import { rpc } from 'luci.rpc';
import { ui } from 'luci.ui';
import { dom } from 'luci.dom';
import { Poll } from 'luci.poll';
```

## Persistence Model

The gateway IP appears in **five places** on the router. The switch script updates them all atomically:

| Location | What | How updated |
|---|---|---|
| Kernel: host route | `<gw_ip> dev br-zt` | `ip route replace` |
| Kernel: table 100 default | `default via <gw_ip> dev br-zt table 100` | `ip route replace` |
| UCI: `/etc/config/network` | `zt_gateway_host` + `zt_gateway_default` routes | `uci set` + `uci commit` |
| Hotplug: `99-zerotier-bridge` | host route + table 100 route + table 1 return | `sed` replace |
| Boot: `/etc/rc.local` | all three routes | `sed` replace |

The hotplug and rc.local currently hardcode `10.11.12.3`. The script replaces these with the new gateway IP. A cleaner future approach would be to have the hotplug/rc.local read from UCI, but that's a refactoring step.

## Health Check

Default method: `ping -c 2 -W 3 <gateway_ip>` over the ZT network.

- Runs every `health_interval` seconds via cron or the LuCI polling mechanism
- If active gateway becomes unreachable AND `auto_failback='1'`: switch to the `default='1'` gateway
- Health status displayed in UI with latency

## Rollback / Safety

- Pre-flight: refuse to switch if new gateway is unreachable
- Graceful drain timeout: if drain doesn't complete, auto-fallback to force
- If switch script fails partway: the host route + table 100 are updated atomically; worst case the old conntrack entries persist and time out naturally
- Manual recovery: `uci set zt-gateway.global.active_gateway='<region>'` + run `/usr/sbin/zt-gateway-switch <ip> force`

## Files to Create

| File | Purpose |
|---|---|
| `luci-app-zt-gateway/Makefile` | OpenWRT build system |
| `luci-app-zt-gateway/root/etc/config/zt-gateway` | UCI config skeleton |
| `luci-app-zt-gateway/root/usr/share/acl.d/luci-app-zt-gateway.json` | RPC + UCI ACL |
| `luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | Menu entry |
| `luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc` | rpcd backend |
| `luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch` | Switch script |
| `luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js` | Frontend UI |

## Verification

1. Install package on router
2. Configure 2+ gateways in `/etc/config/zt-gateway`
3. **Force switch test**: select new region → Force → verify `ip route show table 100` shows new gateway, `curl ifconfig.me` from WIBLAN client shows new exit IP
4. **Graceful switch test**: select new region → Graceful → verify existing SSH on WIBLAN client stays alive, new connections go through new gateway, drain counter decrements
5. **Fail test**: switch to unreachable gateway → verify pre-flight blocks the switch with error message
6. **Drain timeout test**: graceful switch with 30s timeout → verify it falls back to force after timeout
7. **Reboot persistence**: switch gateway → reboot router → verify table 100 still has the new gateway

## Future: ZeroTier Central API Integration

Skipping for now. When added, the switch script should:

1. Call `DELETE /api/network/{networkId}/route/{routeId}` for the old `0.0.0.0/0` managed route
2. Call `POST /api/network/{networkId}/route` with `{ "target": "0.0.0.0/0", "via": "<new_gateway_ip>" }`
3. Toggle "Allow Default Route Override" on new gateway node, disable on old

Requires storing a ZeroTier Central API token in UCI:
```
config global 'global'
    option zt_central_token ''
    option zt_network_id 'e3918db48378cb6a'
```

This is only needed so other ZeroTier peers also get the correct default route — WIBLAN clients are already handled by the local `ip rule`.

## Future: Gateway Auto-Provisioning

Currently, each gateway node must be manually set up with:
- ZeroTier membership + static IP
- IP forwarding + NAT
- iptables FORWARD rules

A future enhancement could SSH into candidate nodes from the controller and set up NAT automatically, or use an agent on each gateway that phones home. Out of scope for v1.

---

## Building and Installing luci-app-zt-gateway

This section covers how to build the `luci-app-zt-gateway` package from source into an `.ipk` using the OpenWRT SDK, then install it on the target router.

### Prerequisites

- A **Linux** development machine (Ubuntu 22.04+ / Debian 12+ recommended)
- **~2 GB** free disk space for the SDK
- Basic build tools: `gcc`, `make`, `perl`, `python3`, `zstd`
- SSH/SCP access to the target router (`192.168.13.1`)
- The target router's **OpenWRT version** and **architecture** (determine these before downloading the SDK)

#### Determine router architecture

SSH into the router and run:

```sh
cat /etc/openwrt_release   # shows OPENWRT_RELEASE, OPENWRT_ARCH, OPENWRT_BOARD
opkg print-architecture    # lists available architectures
```

The `OPENWRT_ARCH` line tells you the target (e.g. `mipsel_24kc`, `aarch64_cortex-a53`, `x86_64`).

### Step 1: Download the OpenWRT SDK

Download the SDK matching your router's **exact OpenWRT version and architecture** from the OpenWRT downloads server:

- **OpenWRT 24.10**: https://downloads.openwrt.org/releases/24.10.7/targets/
- **OpenWRT 23.05**: https://downloads.openwrt.org/releases/23.05.5/targets/

Navigate into the appropriate target subdirectory (e.g. `x86/64/`, `ramips/mt7621/`, `mediatek/filogic/`) and download the `openwrt-sdk-*.tar.zst` (or `.tar.xz`) file.

```sh
# Example for x86_64 / OpenWRT 24.10
wget https://downloads.openwrt.org/releases/24.10.7/targets/x86/64/openwrt-sdk-24.10.7-x86-64_gcc-13.3.0_musl.Linux-x86_64.tar.zst

# Extract
tar --zstd -xf openwrt-sdk-24.10.7-*.tar.zst
cd openwrt-sdk-24.10.7-*
```

### Step 2: Prepare the SDK

```sh
# Update and install standard feed definitions
./scripts/feeds update -a
./scripts/feeds install -a
```

### Step 3: Add the package source

Place the `luci-app-zt-gateway` directory into the SDK's package tree. There are two approaches:

#### Option A: Direct placement (simplest)

```sh
# Copy or symlink the package directory into the SDK
# If your source is at ~/projects/luci-app-zt-gateway/:
ln -s ~/projects/luci-app-zt-gateway package/luci-app-zt-gateway
```

#### Option B: Custom feed (recommended for multiple packages)

```sh
# 1. Create a feed directory
mkdir -p ~/my-openwrt-feed
ln -s ~/projects/luci-app-zt-gateway ~/my-openwrt-feed/luci-app-zt-gateway

# 2. Register the feed in the SDK
echo 'src-link customfeed /home/YOURUSER/my-openwrt-feed' >> feeds.conf.default

# 3. Update and install the custom feed
./scripts/feeds update customfeed
./scripts/feeds install -p customfeed luci-app-zt-gateway
```

### Step 4: The Makefile

The `luci-app-zt-gateway/Makefile` uses the LuCI build system (`luci.mk`) which automates directory mapping, installation, and packaging. The Makefile must be in the package root alongside the `htdocs/` and `root/` directories.

```makefile
# luci-app-zt-gateway/Makefile
include $(TOPDIR)/rules.mk

PKG_NAME:=luci-app-zt-gateway
LUCI_TITLE:=LuCI support for ZeroTier Exit Gateway Switching
LUCI_DEPENDS:=+luci-base +ucode +conntrack
LUCI_PKGARCH:=all
PKG_VERSION:=1.0.0
PKG_RELEASE:=1
PKG_LICENSE:=Apache-2.0
PKG_MAINTAINER:=Your Name <you@example.com>

include $(TOPDIR)/feeds/luci/luci.mk

# call BuildPackage - OpenWrt buildroot signature
```

Key points:

- `luci.mk` (not `package.mk`) is used for LuCI applications. It scans `htdocs/` and `root/` and automatically creates install rules that map files to their target paths:
  - `htdocs/luci-static/resources/view/` → `/www/luci-static/resources/view/`
  - `root/` → mirrors directly onto the target filesystem root (`/`)
- `LUCI_PKGARCH:=all` because LuCI apps are architecture-independent (JS, ucode, shell only).
- `include $(TOPDIR)/feeds/luci/luci.mk` uses the absolute path variant. If building inside the main `openwrt/luci` feed, use `include ../../luci.mk` instead.
- The trailing comment `# call BuildPackage - OpenWrt buildroot signature` is **required** by the OpenWRT feed scanner to discover this package.
- **Dependencies**: `+luci-base` is mandatory. `+ucode` is needed for the rpcd backend. `+conntrack` is needed because the switch script uses `conntrack` CLI.

#### Directory-to-path mapping (luci.mk automatics)

When `luci.mk` processes this package, it automatically installs:

| Source path in package | Target path on router |
|---|---|
| `htdocs/luci-static/resources/view/zt-gateway/overview.js` | `/www/luci-static/resources/view/zt-gateway/overview.js` |
| `root/etc/config/zt-gateway` | `/etc/config/zt-gateway` |
| `root/usr/share/rpcd/ucode/zt-gateway.uc` | `/usr/share/rpcd/ucode/zt-gateway.uc` |
| `root/usr/share/acl.d/luci-app-zt-gateway.json` | `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` |
| `root/usr/share/luci/menu.d/luci-app-zt-gateway.json` | `/usr/share/luci/menu.d/luci-app-zt-gateway.json` |
| `root/usr/sbin/zt-gateway-switch` | `/usr/sbin/zt-gateway-switch` |

### Step 5: Configure and build

```sh
# Select the package in menuconfig
make menuconfig
# Navigate: LuCI → 3. Applications → luci-app-zt-gateway
# Press 'M' to build as module (do NOT press 'Y' — we want an .ipk, not a firmware image)
# Save and exit

# Build the package with verbose output
make package/luci-app-zt-gateway/compile V=s
```

If `make menuconfig` doesn't show the package, ensure the feeds are updated:

```sh
./scripts/feeds update -a
./scripts/feeds install luci-app-zt-gateway
```

### Step 6: Locate the built .ipk

```sh
find bin/ -name 'luci-app-zt-gateway*.ipk' -type f
```

Typical output path: `bin/packages/<architecture>/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk`

Because `LUCI_PKGARCH:=all`, the `.ipk` filename ends with `_all.ipk` — it's architecture-independent and works on any OpenWRT target.

### Step 7: Install on the router

#### Transfer the package

```sh
scp bin/packages/*/customfeed/luci-app-zt-gateway_1.0.0-1_all.ipk root@192.168.13.1:/tmp/
```

#### Install via opkg (OpenWRT ≤ 24.10)

```sh
ssh root@192.168.13.1
opkg update
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```

If you get dependency errors (e.g. missing `luci-base`), install them first:

```sh
opkg install luci-base ucode conntrack
opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```

#### Install via apk (OpenWRT 25.12+)

Starting with OpenWRT 25.12, the package manager is `apk` instead of `opkg`:

```sh
ssh root@192.168.13.1
apk add --allow-untrusted /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```

#### Post-install: refresh LuCI cache

```sh
rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
```

Then navigate to the LuCI web UI — the "ZeroTier Exit Gateway" page should appear under **Services**.

### Step 8: Configure gateways

After installation, edit the UCI config to add your gateway nodes:

```sh
vi /etc/config/zt-gateway
```

Or use `uci` commands:

```sh
uci set zt-gateway.global.active_gateway='amsterdam'
uci set zt-gateway.global.switch_mode='force'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='amsterdam'
uci set zt-gateway.@gateway[-1].label='Amsterdam (ocirosea641)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.3'
uci set zt-gateway.@gateway[-1].default='1'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='tirunelveli'
uci set zt-gateway.@gateway[-1].label='Tirunelveli (rpi1000)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.5'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci add zt-gateway gateway
uci set zt-gateway.@gateway[-1].region='bangalore'
uci set zt-gateway.@gateway[-1].label='Bangalore (sensecap-m4)'
uci set zt-gateway.@gateway[-1].ip='10.11.12.4'
uci set zt-gateway.@gateway[-1].default='0'
uci set zt-gateway.@gateway[-1].health_check='ping'

uci commit zt-gateway
```

### Step 9: Verify the rpcd backend is registered

After install, verify the ubus backend is available:

```sh
# Restart rpcd to pick up the new backend
/etc/init.d/rpcd restart

# List available methods
ubus list zt-gateway.*
# Expected output:
#   zt-gateway.status
#   zt-gateway.switch
#   zt-gateway.health
#   zt-gateway.drain_status
#   zt-gateway.cancel_drain

# Test the status method
ubus call zt-gateway status
```

If `ubus list` doesn't show `zt-gateway.*`, check:

1. The ucode script is executable and at the correct path: `ls -la /usr/share/rpcd/ucode/zt-gateway.uc`
2. The ACL file exists: `cat /usr/share/rpcd/acl.d/luci-app-zt-gateway.json`
3. rpcd log output: `logread -e rpcd`

### Quick-build cheat sheet (no SDK, manual .ipk)

If you don't want to bother with the full SDK and just want to produce an `.ipk` manually, you can use the `opkg-utils` tools on any Linux machine:

```sh
# Install opkg-utils
sudo apt install opkg-utils   # Debian/Ubuntu

# Create the package staging directory
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d
mkdir -p /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin

# Copy source files into staging (adjust source paths as needed)
cp htdocs/luci-static/resources/view/zt-gateway/overview.js \
   /tmp/luci-app-zt-gateway_1.0.0-1_all/www/luci-static/resources/view/zt-gateway/
cp root/etc/config/zt-gateway /tmp/luci-app-zt-gateway_1.0.0-1_all/etc/config/
cp root/usr/share/rpcd/ucode/zt-gateway.uc /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/ucode/
cp root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/rpcd/acl.d/
cp root/usr/share/luci/menu.d/luci-app-zt-gateway.json /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/share/luci/menu.d/
cp root/usr/sbin/zt-gateway-switch /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/
chmod +x /tmp/luci-app-zt-gateway_1.0.0-1_all/usr/sbin/zt-gateway-switch

# Write the control file
cat > /tmp/luci-app-zt-gateway_1.0.0-1_all/CONTROL/control << 'EOF'
Package: luci-app-zt-gateway
Version: 1.0.0-1
Depends: libc, luci-base, ucode, conntrack
Source: luci-app-zt-gateway
Section: luci
Architecture: all
Installed-Size: 20480
Description: LuCI support for ZeroTier Exit Gateway Switching
EOF

# Build the .ipk
cd /tmp
opkg-build -O -o root -g root luci-app-zt-gateway_1.0.0-1_all

# Result: /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```

This manual approach skips the SDK entirely. It's suitable for quick iteration during development. The tradeoff: no automatic dependency resolution, no translation compilation, no JS minification.

### Development workflow: iterate fast without rebuilding

During development, you can skip the build step entirely by editing files directly on the router:

```sh
# Edit the JS view directly
scp htdocs/luci-static/resources/view/zt-gateway/overview.js \
    root@192.168.13.1:/www/luci-static/resources/view/zt-gateway/overview.js

# Edit the ucode backend
scp root/usr/share/rpcd/ucode/zt-gateway.uc \
    root@192.168.13.1:/usr/share/rpcd/ucode/zt-gateway.uc
/etc/init.d/rpcd restart

# Edit the switch script
scp root/usr/sbin/zt-gateway-switch \
    root@192.168.13.1:/usr/sbin/zt-gateway-switch

# Edit the UCI config
scp root/etc/config/zt-gateway \
    root@192.168.13.1:/etc/config/zt-gateway

# Clear LuCI browser cache after JS changes
ssh root@192.168.13.1 'rm -rf /tmp/luci-indexcache /tmp/luci-modulecache'
```

Build the `.ipk` only when you need a clean install or are sharing the package.

### Troubleshooting

| Problem | Cause | Fix |
|---|---|---|
| `make menuconfig` doesn't show the package | Feed not registered | Run `./scripts/feeds update -a && ./scripts/feeds install -a` |
| `opkg install` fails with "cannot install package" | Missing dependencies | Run `opkg update` then install dependencies first |
| LuCI page shows 404 | JS view not found at expected path | Verify `/www/luci-static/resources/view/zt-gateway/overview.js` exists on router |
| ubus methods missing | rpcd didn't pick up the ucode backend | `/etc/init.d/rpcd restart`; check `logread -e rpcd` for errors |
| ACL denied when calling ubus from LuCI | ACL JSON file missing or wrong format | Verify `/usr/share/rpcd/acl.d/luci-app-zt-gateway.json` and restart rpcd |
| `luci.mk` not found during build | Building outside the LuCI feed | Use `include $(TOPDIR)/feeds/luci/luci.mk` (absolute) instead of `include ../../luci.mk` (relative) |
| JS view loads but is blank | Import path errors in JS | Open browser dev console; LuCI JS modules use `luci.view`, `luci.form`, etc. |


## Local Testing with Docker (macvlan)

This section covers how to test `luci-app-zt-gateway` locally using the official `openwrt/rootfs` Docker image with macvlan networking. This gives you a full OpenWRT environment with real `ip rule`, `iptables`, `conntrack`, and rpcd — no physical router required.

### Test architecture

```
┌──────────────────────────────────────────────────────────────────┐
│  Docker host (your dev machine)                                  │
│                                                                  │
│  ┌──────────────────────┐    ┌──────────────────────┐           │
│  │  openwrt-router      │    │  wiblan-client        │           │
│  │  (macvlan 10.11.13.1)│    │  (macvlan 10.11.13.10)│           │
│  │                      │    │                       │           │
│  │  - LuCI on :80       │    │  - curl / ping        │           │
│  │  - rpcd + ucode      │    │  - traffic exits via  │           │
│  │  - zt-gateway-switch │    │    router's table 100 │           │
│  │  - br-zt + ip rules  │    │                       │           │
│  └──────────┬───────────┘    └───────────┬───────────┘           │
│             │                            │                       │
│             └──────────┬─────────────────┘                       │
│                        │                                         │
│               zt-gateway-macvlan                                 │
│             10.11.13.0/24 subnet                                 │
│             (macvlan bridge mode)                                 │
│                                                                  │
│  ┌──────────────────────┐                                        │
│  │  zt-gw-amsterdam     │  ← Simulates a remote ZeroTier exit   │
│  │  (macvlan 10.11.12.3) │    node with NAT + ip_forward         │
│  │                      │                                        │
│  │  - ip_forward=1      │                                        │
│  │  - iptables MASQ     │                                        │
│  └──────────────────────┘                                        │
│                                                                  │
│               zt-exit-macvlan                                    │
│             10.11.12.0/24 subnet                                 │
└──────────────────────────────────────────────────────────────────┘
```

The key insight: we use **two macvlan networks** to simulate the two network segments the real router has:
1. **zt-gateway-macvlan** (`10.11.13.0/24`) — represents the WIBLAN subnet. The router sits at `.1`, client at `.10`.
2. **zt-exit-macvlan** (`10.11.12.0/24`) — represents the ZeroTier network where exit gateway nodes live.

### Step 1: Create the Dockerfile

Create a Dockerfile that builds a pre-configured OpenWRT image with all dependencies and the zt-gateway package:

```dockerfile
# Dockerfile.openwrt-dev
FROM openwrt/rootfs:x86-64-24.10.7

# Install required packages
RUN opkg update && opkg install \
    luci \
    luci-base \
    ucode \
    conntrack \
    ip-bridge \
    iptables \
    kmod-ipt-conntrack \
    kmod-ipt-connmark \
    kmod-ipt-mark \
    kmod-ipt-extra \
    && rm -rf /var/opkg-lists/*

# Enable uhttpd
RUN /etc/init.d/uhttpd enable
RUN /etc/init.d/uhttpd start

# Enable rpcd
RUN /etc/init.d/rpcd enable
RUN /etc/init.d/rpcd start

# Set root password (required for LuCI login)
RUN echo "root:root" | chpasswd

EXPOSE 80

CMD ["/sbin/init"]
```

Build it:

```sh
docker build -t openwrt-zt-gateway-dev -f Dockerfile.openwrt-dev .
```

**Note:** The exact `kmod-*` packages available depend on the OpenWRT version. If `opkg install` fails on kernel modules, skip them — the Docker container's kernel is the host kernel, so the modules are already loaded. What matters is that `iptables -t mangle` and `conntrack` work. Verify with:

```sh
# Run a quick check after building
docker run --rm --privileged openwrt-zt-gateway-dev sh -c \
  "iptables -t mangle -L && echo 'mangle OK' && conntrack -L 2>/dev/null && echo 'conntrack OK'"
```

### Step 2: Create the Docker Compose file

```yaml
# docker-compose.yml
version: "3.8"

networks:
  # Simulates the WIBLAN client network (10.11.13.0/24)
  zt-gateway-lan:
    driver: macvlan
    driver_opts:
      mode: bridge
    ipam:
      config:
        - subnet: "10.11.13.0/24"
          gateway: "10.11.13.1"
          ip_range: "10.11.13.100/28"

  # Simulates the ZeroTier exit node network (10.11.12.0/24)
  zt-exit-net:
    driver: macvlan
    driver_opts:
      mode: bridge
    ipam:
      config:
        - subnet: "10.11.12.0/24"
          ip_range: "10.11.12.100/28"

services:
  # The OpenWRT router running luci-app-zt-gateway
  openwrt-router:
    image: openwrt-zt-gateway-dev
    container_name: openwrt-router
    privileged: true
    networks:
      zt-gateway-lan:
        ipv4_address: "10.11.13.1"
      zt-exit-net:
        ipv4_address: "10.11.12.1"
    ports:
      - "8080:80"
    volumes:
      # Hot-mount source files for live iteration (key advantage)
      - ./luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js:/www/luci-static/resources/view/zt-gateway/overview.js
      - ./luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc:/usr/share/rpcd/ucode/zt-gateway.uc
      - ./luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch:/usr/sbin/zt-gateway-switch
      - ./luci-app-zt-gateway/root/etc/config/zt-gateway:/etc/config/zt-gateway
      - ./luci-app-zt-gateway/root/usr/share/rpcd/acl.d/luci-app-zt-gateway.json:/usr/share/rpcd/acl.d/luci-app-zt-gateway.json
      - ./luci-app-zt-gateway/root/usr/share/luci/menu.d/luci-app-zt-gateway.json:/usr/share/luci/menu.d/luci-app-zt-gateway.json
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.send_redirects=0
    command: ["/sbin/init"]

  # A WIBLAN client to test routing through the gateway
  wiblan-client:
    image: alpine:latest
    container_name: wiblan-client
    networks:
      zt-gateway-lan:
        ipv4_address: "10.11.13.10"
    cap_add:
      - NET_ADMIN
    command: ["sleep", "infinity"]

  # Simulates the amsterdam ZeroTier exit gateway (10.11.12.3)
  zt-gw-amsterdam:
    image: alpine:latest
    container_name: zt-gw-amsterdam
    networks:
      zt-exit-net:
        ipv4_address: "10.11.12.3"
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
    command: ["sleep", "infinity"]

  # Simulates the tirunelveli ZeroTier exit gateway (10.11.12.5)
  zt-gw-tirunelveli:
    image: alpine:latest
    container_name: zt-gw-tirunelveli
    networks:
      zt-exit-net:
        ipv4_address: "10.11.12.5"
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.ip_forward=1
    command: ["sleep", "infinity"]
```

### Step 3: Launch and configure the environment

```sh
# Start all containers
docker compose up -d

# Wait a few seconds for init, then configure the simulated exit gateways
# These simulate real ZeroTier exit nodes with NAT
```

#### Configure the amsterdam exit gateway (10.11.12.3)

```sh
docker exec -it zt-gw-amsterdam sh -c '
  # Enable forwarding
  echo 1 > /proc/sys/net/ipv4/ip_forward

  # Add a default route back to the router so return traffic works
  ip route add 10.11.13.0/24 via 10.11.12.1

  # Add NAT so traffic from WIBLAN clients can reach the internet
  # (or just other container networks for testing)
  apk add --no-cache iptables
  iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
  iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
  iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT

  # Add a fake "internet" endpoint (just the Docker host gateway)
  # In real use, this would be the actual internet
  ping -c 1 10.11.12.1 2>/dev/null
'
```

#### Configure the tirunelveli exit gateway (10.11.12.5)

```sh
docker exec -it zt-gw-tirunelveli sh -c '
  echo 1 > /proc/sys/net/ipv4/ip_forward
  ip route add 10.11.13.0/24 via 10.11.12.1
  apk add --no-cache iptables
  iptables -t nat -A POSTROUTING -s 10.11.13.0/24 -j MASQUERADE
  iptables -A FORWARD -s 10.11.13.0/24 -j ACCEPT
  iptables -A FORWARD -d 10.11.13.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
'
```

#### Configure the OpenWRT router

```sh
docker exec -it openwrt-router sh
```

Inside the router container:

```sh
# 1. Create the br-zt bridge (simulates the ZeroTier bridge)
ip link add name br-zt type bridge
ip link set br-zt up

# Assign the ZT-exit IP to the bridge so the router can reach 10.11.12.0/24
ip addr add 10.11.12.1/24 dev br-zt

# 2. Set up policy routing (same as the real router)

# Host route to the current gateway (ARP fix)
ip route replace 10.11.12.3 dev br-zt

# Default route via the current gateway in table 100
ip route replace default via 10.11.12.3 dev br-zt table 100

# Route WIBLAN subnet via br-zt in mwan3 table (for return traffic)
ip route replace 10.11.13.0/24 dev br-zt table 1

# Source-based routing rule: WIBLAN traffic → table 100
ip rule add from 10.11.13.0/24 table 100 priority 100

# 3. Verify routing
ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt

ip rule show
# Expected: 100: from 10.11.13.0/24 lookup 100

# 4. Test from the router itself
ping -c 2 10.11.12.3   # Should reach amsterdam gateway
```

#### Configure the WIBLAN client

```sh
docker exec -it wiblan-client sh -c '
  # Add a route to the router for the ZT subnet
  ip route add 10.11.12.0/24 via 10.11.13.1

  # Set the OpenWRT router as default gateway
  ip route add default via 10.11.13.1

  # Test connectivity
  ping -c 2 10.11.12.3
'
```

### Step 4: Install and test the luci-app-zt-gateway package

#### Option A: The package is already volume-mounted

If you used the `volumes:` section in `docker-compose.yml`, the files are already in place. Just restart services:

```sh
docker exec -it openwrt-router sh -c '
  chmod +x /usr/sbin/zt-gateway-switch
  /etc/init.d/rpcd restart
  rm -rf /tmp/luci-indexcache /tmp/luci-modulecache
'
```

#### Option B: Install the .ipk

If you built an `.ipk`, copy it into the container:

```sh
docker cp luci-app-zt-gateway_1.0.0-1_all.ipk openwrt-router:/tmp/
docker exec -it openwrt-router opkg install /tmp/luci-app-zt-gateway_1.0.0-1_all.ipk
```

#### Verify the rpcd backend

```sh
docker exec -it openwrt-router ubus list zt-gateway.*
# Expected:
#   zt-gateway.status
#   zt-gateway.switch
#   zt-gateway.health
#   zt-gateway.drain_status
#   zt-gateway.cancel_drain

docker exec -it openwrt-router ubus call zt-gateway status
```

#### Access the LuCI UI

Open http://localhost:8080 in your browser. Log in with `root` / `root`. Navigate to **Services → ZeroTier Exit Gateway**.

### Step 5: Run the test scenarios

#### Force switch test

```sh
# 1. Verify current gateway is amsterdam (10.11.12.3)
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.3 dev br-zt

# 2. Trigger a force switch to tirunelveli via the CLI script
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force

# 3. Verify the route changed
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt

# 4. Verify conntrack was flushed
docker exec -it openwrt-router conntrack -L -s 10.11.13.0/24 2>/dev/null
# Expected: empty (all entries flushed)

# 5. Verify from the client side
docker exec -it wiblan-client ping -c 2 10.11.12.5
```

#### Graceful drain test

```sh
# 1. Start a long-running connection from the client
docker exec -d wiblan-client ping 10.11.12.3   # background ping

# 2. Trigger a graceful switch (this will be done from the LuCI UI
#    or via ubus call)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 300

# 3. Verify both routing tables exist during drain
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt  (new gateway)

docker exec -it openwrt-router ip route show table 101
# Expected: default via 10.11.12.3 dev br-zt  (old gateway, drain)

# 4. Verify the fwmark rule exists
docker exec -it openwrt-router ip rule show
# Expected: 99: from all fwmark 0x100 lookup 101

# 5. Verify the mangle rules
docker exec -it openwrt-router iptables -t mangle -L -v

# 6. Watch drain progress
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l

# 7. Wait for drain to complete (or kill the background ping to speed it up)
docker exec -it wiblan-client pkill ping

# 8. After drain completes, verify cleanup
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain table removed)
docker exec -it openwrt-router ip rule show
# Expected: no fwmark 0x100 rule
```

#### Switch to unreachable gateway test

```sh
# 1. Stop the tirunelveli gateway to simulate it being down
docker stop zt-gw-tirunelveli

# 2. Try to switch to it (should fail pre-flight)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force
# Expected: "Gateway unreachable" error, no route changes

# 3. Verify routes unchanged
docker exec -it openwrt-router ip route show table 100

# 4. Restart for subsequent tests
docker start zt-gw-tirunelveli
```

#### Drain timeout test

```sh
# 1. Start a persistent connection
docker exec -d wiblan-client ping 10.11.12.3

# 2. Graceful switch with short timeout (30s)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 graceful 30

# 3. Watch the drain monitor — it will hit the 30s timeout
#    because the ping keeps conntrack entries alive
docker exec -it openwrt-router conntrack -L -m 0x100 2>/dev/null | wc -l

# 4. After 30s, should auto-fallback to force switch
docker exec -it openwrt-router ip route show table 100
# Expected: default via 10.11.12.5 dev br-zt (force applied)
docker exec -it openwrt-router ip route show table 101
# Expected: empty (drain cleaned up by force)

# 5. Clean up
docker exec -it wiblan-client pkill ping
```

### Step 6: Iterate on fixes

The key advantage of the Docker setup is the **volume mounts** in the compose file. When you edit a source file on the host, it's immediately reflected inside the container.

#### Live-editing workflow

```sh
# 1. Edit a source file on the host
vim luci-app-zt-gateway/root/usr/sbin/zt-gateway-switch

# 2. Re-test inside the container (file is already updated via volume mount)
docker exec -it openwrt-router zt-gateway-switch 10.11.12.5 force

# 3. Check results
docker exec -it openwrt-router ip route show table 100
```

For LuCI JS changes, you need to clear the cache:

```sh
# Edit JS on host
vim luci-app-zt-gateway/htdocs/luci-static/resources/view/zt-gateway/overview.js

# Clear cache inside container
docker exec -it openwrt-router rm -rf /tmp/luci-indexcache /tmp/luci-modulecache

# Refresh browser
```

For ucode backend changes:

```sh
# Edit ucode on host
vim luci-app-zt-gateway/root/usr/share/rpcd/ucode/zt-gateway.uc

# Restart rpcd to reload the backend
docker exec -it openwrt-router /etc/init.d/rpcd restart
```

#### Quick-snapshot the exact container state

If you want to save a known-good state before making a risky change:

```sh
# Save container state as a Docker image
docker commit openwrt-router openwrt-zt-gateway-snapshot:good

# ... make changes, break things ...

# Restore from snapshot
docker compose down
docker run -d --name openwrt-router \
  --privileged \
  -p 8080:80 \
  openwrt-zt-gateway-snapshot:good
```

#### Watch logs in real-time

```sh
# OpenWRT system log
docker exec -it openwrt-router logread -f

# rpcd errors specifically
docker exec -it openwrt-router logread -f -e rpcd

# conntrack events in real-time
docker exec -it openwrt-router conntrack -E
```

#### Run the full test suite in one shot

```sh
#!/bin/bash
# test-zt-gateway.sh — Run all test scenarios in Docker
set -e

ROUTER="openwrt-router"
CLIENT="wiblan-client"
GW_AMS="10.11.12.3"
GW_TIR="10.11.12.5"

echo "=== Test 1: Force switch amsterdam → tirunelveli ==="
docker exec $ROUTER zt-gateway-switch $GW_TIR force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo "  Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_TIR" && echo "  PASS" || echo "  FAIL"

echo "=== Test 2: Force switch tirunelveli → amsterdam ==="
docker exec $ROUTER zt-gateway-switch $GW_AMS force
ROUTE=$(docker exec $ROUTER ip route show table 100 | grep default)
echo "  Route after force: $ROUTE"
echo "$ROUTE" | grep -q "$GW_AMS" && echo "  PASS" || echo "  FAIL"

echo "=== Test 3: Switch to unreachable gateway ==="
docker stop zt-gw-tirunelveli
if docker exec $ROUTER zt-gateway-switch $GW_TIR force 2>&1 | grep -qi "unreachable\|fail\|error"; then
  echo "  PASS (blocked as expected)"
else
  echo "  FAIL (should have been blocked)"
fi
docker start zt-gw-tirunelveli
sleep 3

echo "=== Test 4: Verify client can reach gateway ==="
docker exec $CLIENT ping -c 2 $GW_AMS && echo "  PASS" || echo "  FAIL"

echo "=== Test 5: Verify ubus backend ==="
docker exec $ROUTER ubus call zt-gateway status | grep -q "amsterdam" && echo "  PASS" || echo "  FAIL"

echo "=== All tests complete ==="
```

### Important limitations of the Docker test environment

| Limitation | Impact | Mitigation |
|---|---|---|
| Docker shares the host kernel | `kmod-*` packages from opkg won't load (wrong kernel version) | Use host kernel modules; verify with `lsmod`/`iptables -t mangle -L` |
| `br-zt` is simulated, not a real ZeroTier bridge | No actual ZT membership, no `ztk4jpk77j` interface | Manually create `br-zt` bridge and assign IPs; skip hotplug scripts |
| `conntrack -D -s` may behave differently | Docker's conntrack table includes host traffic too | Use `-s` and `-d` filters; or run `conntrack -F` for a clean slate |
| No real `rc.local` / hotplug execution on boot | Persistence scripts can't be tested end-to-end | Test the switch script logic; verify `sed` replacements on the files separately |
| `iptables -t mangle` requires `NET_ADMIN` | Container must run `--privileged` or `--cap-add=NET_ADMIN` | Already configured in compose file |
| Macvlan host isolation | Host can't ping macvlan containers by default | Use a macvlan shim interface (see below) or just `docker exec` |
| No real ZeroTier Central API | Can't test the future ZT API integration | Mock the API calls or skip this test |

### Fixing the macvlan host-isolation issue

By default, the Docker host **cannot** communicate directly with containers on a macvlan network. If you need to ping/curl the router or client from your host machine (not just via `docker exec`), create a macvlan shim interface:

```sh
# Create a macvlan interface on the host that bridges to the same parent
# Replace eth0 with your host's primary interface
sudo ip link add link eth0 name mv-shim type macvlan mode bridge
sudo ip addr add 10.11.13.254/32 dev mv-shim
sudo ip link set mv-shim up

# Add a route to the WIBLAN subnet via the shim
sudo ip route add 10.11.13.0/24 dev mv-shim

# Now you can reach the containers from host
ping 10.11.13.1        # the router
curl http://10.11.13.1  # LuCI directly (skip the port mapping)

# For the ZT exit subnet
sudo ip link add link eth0 name mv-exit-shim type macvlan mode bridge
sudo ip addr add 10.11.12.254/32 dev mv-exit-shim
sudo ip link set mv-exit-shim up
sudo ip route add 10.11.12.0/24 dev mv-exit-shim
```

Clean up when done:

```sh
sudo ip link del mv-shim
sudo ip link del mv-exit-shim
```

### Teardown

```sh
# Stop and remove all containers
docker compose down

# Remove the shim interfaces (if created)
sudo ip link del mv-shim 2>/dev/null
sudo ip link del mv-exit-shim 2>/dev/null

# Remove the custom images (optional)
docker rmi openwrt-zt-gateway-dev openwrt-zt-gateway-snapshot:good
```