fix: routing ip rule missing after reboot, busybox compat, skill restructure

Routing fixes (2026-07-14):
- Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case
- Add UCI network rule persistence so netifd restores it on boot
- Verify ip rule exists in zt-gateway-switch do_force/do_graceful
- Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken
  awk-over-ip pipeline (busybox awk mishandles exit in compound if)
- Validate bridge port candidate exists as network interface
- Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP
  when enslaved to bridge, causing 'Nexthop has invalid gateway')
- Replace ip rule replace (GNU-only) with del+add for busybox

Infrastructure:
- Fix deploy:install stdin starvation: ssh/scp consume pipe data in
  find|while loop; add </dev/null to prevent truncation
- Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink
- Add policy routing and busybox gotcha sections to SKILL.md
- Add diagnostics doc for the routing fix session

.agents/skills/luci-dev

@@ -0,0 +1 @@
+../../skills/luci-dev

docs/PROGRESS.md

@@ -153,3 +153,42 @@ After installing `luci-compat`, login and page rendering work correctly, but **f
 - ZeroTier requires "Allow Ethernet Bridging" for L2 traffic
 - nftables fw4 zones must explicitly include bridge interfaces
 - Policy routing table 100 must route via exit gateway, not local IP
+## Date: 2026-07-14
+
+## Post-Reboot Routing Fixes
+
+### Problem
+
+After rebooting the production OpenWrt router, WiBLAN clients
+(10.11.13.x) could not route traffic through the active exit
+gateway (Amsterdam, 10.11.12.3). The `ip rule` directing WIBLAN
+traffic to policy table 100 was missing entirely.
+
+### Root Cause
+
+Three gaps in the hotplug/setup-routing/switch pipeline:
+1. Hotplug `ifup` case never installed the `from 10.11.13.0/24` ip rule
+2. UCI persistence wrote routes but not the `network rule` section
+3. `zt-gateway-switch` never verified the rule existed
+
+### Fixes
+
+1. Added ip rule to hotplug heredoc (`zt-gateway-setup`)
+2. Added `network.zt_wiblan_rule` UCI section to `setup-routing` persistence
+3. Added rule verification to `do_force()`/`do_graceful()` in `zt-gateway-switch`
+
+### Additional fixes during the same session
+
+- **BRIDGE_PORTS auto-detect**: Replaced broken awk-over-ip pipeline
+  with UCI lookup + `/proc/net/dev` fallback + interface existence
+  validation (busybox compatibility)
+- **Route dev param**: Changed `dev $BRIDGE_PORTS` to `dev $BRIDGE`
+  in `setup-routing` (ZT interface has no IP when enslaved to bridge)
+- **`ip rule replace`**: Replaced GNU-only `ip rule replace` with
+  `ip rule del` + `ip rule add` for busybox compatibility
+- **Deploy stdin starvation**: Added `</dev/null` to `ssh`/`scp` in
+  `deploy:install` pipe loop
+- **Deploy overwriting UCI**: Noted that `deploy:install` overwrites
+  `/etc/config/zt-gateway` with repo version, clobbering production
+  customizations (e.g. real Amsterdam IP 10.11.12.3 vs placeholder
+  10.99.12.3)

docs/diagnostics/zerotier-wiblan-missing-iprule-2026-07-14.md

@@ -0,0 +1,124 @@
+# ZeroTier Gateway Routing Fixes
+
+## Date: 2026-07-14
+
+## Problem
+
+After switching to the Amsterdam node, traffic from WiBLAN AP clients
+(10.11.13.0/24) was not routed through the Amsterdam exit gateway
+(10.11.12.3). The table 100 default route pointed correctly to
+Amsterdam, but traffic never reached it.
+
+## Root Cause
+
+The `ip rule` directing WIBLAN subnet traffic to policy table 100 was
+missing:
+
+```
+100: from 10.11.13.0/24 lookup 100
+```
+
+Without this rule, WIBLAN client traffic fell through to the main
+routing table (priority 32766) which routed via WAN, bypassing the
+ZeroTier tunnel entirely.
+
+### Why it was missing
+
+1. **Hotplug script omission**: The hotplug `ifup` case in
+   `zt-gateway-setup` installed table 100/101 routes and the fwmark
+   drain rule, but never installed the `from 10.11.13.0/24` source
+   rule.
+
+2. **No UCI persistence**: `setup-routing` installed the rule at
+   runtime but its UCI persistence block only wrote route sections,
+   not a `network rule` section. So netifd couldn't restore it on
+   boot.
+
+3. **Switch script gap**: `zt-gateway-switch` (`do_force`/`do_graceful`)
+   assumed the rule already existed and never verified or re-added it.
+
+## Fixes Applied
+
+### 1. Hotplug script (`zt-gateway-setup` heredoc)
+
+Added to the `ifup` case after the table routes:
+
+```sh
+ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
+    ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
+```
+
+### 2. UCI persistence (`setup-routing`)
+
+Added a `network rule` UCI section so netifd restores the rule on boot:
+
+```sh
+uci -q set "network.zt_wiblan_rule=rule"
+uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
+uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
+uci -q set "network.zt_wiblan_rule.priority=100"
+```
+
+### 3. Switch script (`zt-gateway-switch`)
+
+Added rule verification to both `do_force()` and `do_graceful()`:
+
+```sh
+ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
+    ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
+```
+
+## Additional Bugs Fixed
+
+### BRIDGE_PORTS auto-detection failure (busybox)
+
+The `awk` pipeline over `ip -o link show` failed silently on busybox
+because busybox awk mishandles `exit` inside compound `if` blocks in
+`-F` pipelines. Fixed by:
+
+1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports`
+2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'`
+3. Validating the result exists: `ip link show "$candidate"`
+
+### Route dev parameter
+
+`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for
+default routes, but the ZT interface has no IP after being enslaved
+to the bridge. Changed to `dev $BRIDGE` (the bridge device that
+holds the 10.11.12.x/23 subnet IP).
+
+### `ip rule replace` busybox incompatibility
+
+Busybox `ip` does not support `ip rule replace`. Replaced all
+occurrences with:
+
+```sh
+ip rule del ... 2>/dev/null || true
+ip rule add ...
+```
+
+### Deploy script stdin starvation
+
+The `mise run deploy:install` script used `find | sort | while read`
+but `ssh`/`scp` inside the loop consumed stdin from the pipe,
+starving the `while read` after the first file. Fixed by adding
+`</dev/null` to `ssh` and `scp` commands in the loop.
+
+## Verification Commands
+
+```bash
+# Check ip rule exists
+ip rule show | grep "lookup 100"
+# Expected: 100: from 10.11.13.0/24 lookup 100
+
+# Check table 100 routes
+ip route show table 100
+# Expected: default via <active_gw_ip> dev br-zt
+
+# Check UCI rule persisted
+uci show network | grep zt_wiblan_rule
+# Expected: network.zt_wiblan_rule=rule, src, lookup, priority
+
+# Ping exit gateway via bridge
+ping -c 2 -I br-zt <active_gw_ip>
+```

mise.toml

@@ -98,8 +98,8 @@ find root/ htdocs/ -type f | sort | while IFS= read -r src; do
         *) echo "SKIP: $src (unknown prefix)"; continue ;;
     esac
     destdir="${dest%/*}"
-    ssh -q "$HOST" mkdir -p "$destdir"
+    ssh -q "$HOST" mkdir -p "$destdir" </dev/null
-    scp -O -q "$src" "$HOST:$dest"
+    scp -O -q "$src" "$HOST:$dest" </dev/null
     echo "    ${src#./}"
     UPLOADED=$((UPLOADED + 1))
 done

root/usr/sbin/zt-gateway-setup

@@ -52,8 +52,20 @@ BRIDGE="${ZTG_BRIDGE:-br-zt}"
 if [ -n "${ZTG_BRIDGE_PORTS:-}" ]; then
 	BRIDGE_PORTS="$ZTG_BRIDGE_PORTS"
 else
-	BRIDGE_PORTS=$(ip -o link show 2>/dev/null \
+	# Auto-detect: try UCI first, then /proc/net/dev (busybox-safe),
-		| awk -F': ' '/^[0-9]+:/{gsub(/@.*/, "", $2); if ($2 ~ /^zt/ && $2 != "br-zt") print $2; exit}')
+	# then ip link as last resort. Busybox awk mishandles `exit` inside
+	# compound `if` blocks in `-F` pipelines, so avoid that pattern.
+	# Validate that any candidate actually exists as a network interface.
+	BRIDGE_PORTS=$(uci -q get zt-gateway.global.bridge_ports 2>/dev/null || true)
+	if [ -n "$BRIDGE_PORTS" ] && ! ip link show "$BRIDGE_PORTS" >/dev/null 2>&1; then
+		BRIDGE_PORTS=""
+	fi
+	if [ -z "$BRIDGE_PORTS" ]; then
+		BRIDGE_PORTS=$(awk -F': ' '/^zt/{print $1; exit}' /proc/net/dev 2>/dev/null || true)
+	fi
+	if [ -z "$BRIDGE_PORTS" ]; then
+		BRIDGE_PORTS=$(ip -br link 2>/dev/null | awk '/^zt/ && !/br-zt/{print $1; exit}')
+	fi
 	if [ -z "$BRIDGE_PORTS" ]; then
 		BRIDGE_PORTS="ztabc0"
 	fi
@@ -309,11 +321,11 @@ cmd_setup_routing() {
 	fi
 	if [ -n "$EXIT_GW" ]; then
 		log "exit gateway: ${EXIT_GW}"
-		ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_MAIN"
+		ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_MAIN"
 		ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MAIN"
 
 		# Table 101 (drain): default via exit gateway
-		ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_DRAIN"
+		ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_DRAIN"
 	else
 		log "warning: could not determine exit gateway; using WIBLAN_GW"
 		ip route replace default via "$WIBLAN_GW" dev "$BRIDGE" table "$TABLE_MAIN"
@@ -325,12 +337,12 @@ cmd_setup_routing() {
 	ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MWAN"
 
 	# ip rule: WIBLAN subnet -> main policy table
-	ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
+	ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
-		ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
+	ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
 
 	# ip rule: fwmark 0x100 -> drain table
-	ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \
+	ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true
-		ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
+	ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
 
 	log "routing setup complete"
 
@@ -362,6 +374,14 @@ cmd_setup_routing() {
 			uci -q set "network.zt_wiblan_subnet.interface=${BRIDGE}"
 			uci -q set "network.zt_wiblan_subnet.table=${TABLE_MAIN}"
 		fi
+		# Policy rule: WIBLAN subnet → main policy table
+		if ! uci -q get "network.zt_wiblan_rule" >/dev/null 2>&1; then
+			uci -q set "network.zt_wiblan_rule=rule"
+			uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
+			uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
+			uci -q set "network.zt_wiblan_rule.priority=100"
+		fi
+
 		uci commit network
 		log "routing UCI config committed"
 	fi
@@ -542,11 +562,15 @@ case "$ACTION" in
 
 		# mwan3 return
 		ip route replace "$ZTG_WIBLAN_CIDR" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MWAN"
+		# ip rule: WIBLAN subnet → main policy table
+		ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || true
+		ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
+
 
 		# ip rule for drain fwmark
+		ip rule del fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
+			priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || true
 		ip rule add fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
-			priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || \
-			ip rule replace fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
 			priority "$ZTG_DRAIN_PRIORITY"
 		;;
 	ifdown)

root/usr/sbin/zt-gateway-switch

@@ -199,6 +199,10 @@ do_force() {
 
 	preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
 
+	# Ensure WIBLAN traffic uses the policy table (may be missing after reboot)
+	ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
+	ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
+
 	set_host_route "$new_ip"
 	set_table_default "$new_ip" "$TABLE_MAIN"
 	ensure_mwan_return
@@ -249,8 +253,8 @@ drain_install_rules() {
 	old_ip=$1
 	ip route replace default via "$old_ip" dev "$BRIDGE" table "$TABLE_DRAIN"
 	mangle_rules_install
-	ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \
+	ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true
-		ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
+	ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
 }
 
 drain_cleanup() {
@@ -353,6 +357,10 @@ do_graceful() {
 	fi
 
 	preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
+	# Ensure WIBLAN traffic uses the policy table (may be missing after reboot)
+	ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
+	ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
+
 
 	printf '%s\n' "$new_ip" >"$DRAIN_NEWFILE"
 	printf '%s\n' "$old_ip" >"$DRAIN_OLDFILE"

skills/SKILL.md

@@ -179,3 +179,86 @@ Some OpenWrt packages need surgical patches for containerized environments:
   let globals = proto({ include: (name, args) => self.render_any(name, args ?? {}) }, scope ?? {});
   ```
 - **`/usr/share/rpcd/ucode/system.uc`**: Only needed if `rpcd-mod-iwinfo` is absent. `luci-mod-admin-full` depends on `rpcd-mod-iwinfo`, so in typical LuCI installs `system.board` is natively available.
+
+
+## OpenWrt Policy Routing (ZeroTier Exit Gateway)
+
+Policy routing for subnet traffic (e.g. WIBLAN 10.11.13.0/24) requires
+three components working together:
+
+### 1. UCI Network Rule (persists across reboot)
+
+```uci
+config rule
+    option src '10.11.13.0/24'
+    option lookup '100'
+    option priority '100'
+```
+
+This is the **only** component that netifd restores automatically on boot.
+Routes in custom tables are also restored, but the `ip rule` that directs
+traffic TO those tables must be explicitly defined as a UCI `network rule`.
+
+### 2. Policy Table Routes
+
+Table 100 (main policy) and 101 (drain) must have default routes via the
+active exit gateway:
+
+```sh
+ip route replace default via <gateway_ip> dev br-zt table 100
+ip route replace default via <gateway_ip> dev br-zt table 101
+```
+
+**Critical**: Use the bridge device (`br-zt`), not the raw ZeroTier
+interface (`ztk4jpk77j`). After the ZT interface is enslaved to the
+bridge, it has no IP and `ip route replace default via <gw> dev <zt_if>`
+fails with `Nexthop has invalid gateway`.
+
+### 3. Hotplug Script
+
+Re-applies routes when the bridge comes up (e.g. after ZeroTier restart).
+Must install BOTH the table routes AND the ip rule:
+
+```sh
+# Table routes
+ip route replace default via "$active_ip" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MAIN"
+# Ip rule (commonly forgotten!)
+ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
+    ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null
+ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
+```
+
+### Verification
+
+```bash
+# Rule exists?
+ip rule show | grep "lookup 100"
+# Table has correct default?
+ip route show table 100
+# Gateway reachable via bridge?
+ping -c 2 -I br-zt <gateway_ip>
+```
+
+## Busybox / OpenWrt Shell Gotchas
+
+1. **`ip rule replace` does not exist** in busybox `ip`. Use
+   `ip rule del ... 2>/dev/null || true; ip rule add ...` instead.
+
+2. **`awk exit` in compound if blocks**: Busybox awk mishandles `exit`
+   inside `if (...) { ...; exit }` when used with `-F` field separator
+   in a pipeline. Workaround: use `/proc/net/dev` as input instead of
+   piping from `ip -o link show`, or use multi-line awk programs.
+
+3. **Shell pipe + ssh stdin starvation**: `ssh` and `scp` consume stdin.
+   In `find | sort | while read; do ssh ...; done` loops, add `</dev/null`
+   to `ssh`/`scp` commands to prevent them from eating the pipe data.
+
+4. **UCI `network rule` vs runtime `ip rule add`**: `ip rule add` only
+   persists in the running kernel. For boot survival, write a UCI
+   `network rule` section so netifd applies it. But note: netifd may
+   not handle `ip rule replace` idempotency — always `del`+`add`.
+
+5. **Bridge ports auto-detection**: Don't rely on `ip -o link show | awk`
+   for interface detection on busybox. Prefer reading `/proc/net/dev` or
+   `/sys/class/net/` which are simpler and more portable. Always validate
+   the detected interface actually exists with `ip link show <name>`.