fix: routing ip rule missing after reboot, busybox compat, skill restructure

Routing fixes (2026-07-14):
- Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case
- Add UCI network rule persistence so netifd restores it on boot
- Verify ip rule exists in zt-gateway-switch do_force/do_graceful
- Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken
  awk-over-ip pipeline (busybox awk mishandles exit in compound if)
- Validate bridge port candidate exists as network interface
- Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP
  when enslaved to bridge, causing 'Nexthop has invalid gateway')
- Replace ip rule replace (GNU-only) with del+add for busybox

Infrastructure:
- Fix deploy:install stdin starvation: ssh/scp consume pipe data in
  find|while loop; add </dev/null to prevent truncation
- Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink
- Add policy routing and busybox gotcha sections to SKILL.md
- Add diagnostics doc for the routing fix session
This commit is contained in:
2026-07-14 18:01:30 +05:30
parent 10081b33e2
commit 581d625044
9 changed files with 294 additions and 15 deletions

View File

@@ -153,3 +153,42 @@ After installing `luci-compat`, login and page rendering work correctly, but **f
- ZeroTier requires "Allow Ethernet Bridging" for L2 traffic
- nftables fw4 zones must explicitly include bridge interfaces
- Policy routing table 100 must route via exit gateway, not local IP
## Date: 2026-07-14
## Post-Reboot Routing Fixes
### Problem
After rebooting the production OpenWrt router, WiBLAN clients
(10.11.13.x) could not route traffic through the active exit
gateway (Amsterdam, 10.11.12.3). The `ip rule` directing WIBLAN
traffic to policy table 100 was missing entirely.
### Root Cause
Three gaps in the hotplug/setup-routing/switch pipeline:
1. Hotplug `ifup` case never installed the `from 10.11.13.0/24` ip rule
2. UCI persistence wrote routes but not the `network rule` section
3. `zt-gateway-switch` never verified the rule existed
### Fixes
1. Added ip rule to hotplug heredoc (`zt-gateway-setup`)
2. Added `network.zt_wiblan_rule` UCI section to `setup-routing` persistence
3. Added rule verification to `do_force()`/`do_graceful()` in `zt-gateway-switch`
### Additional fixes during the same session
- **BRIDGE_PORTS auto-detect**: Replaced broken awk-over-ip pipeline
with UCI lookup + `/proc/net/dev` fallback + interface existence
validation (busybox compatibility)
- **Route dev param**: Changed `dev $BRIDGE_PORTS` to `dev $BRIDGE`
in `setup-routing` (ZT interface has no IP when enslaved to bridge)
- **`ip rule replace`**: Replaced GNU-only `ip rule replace` with
`ip rule del` + `ip rule add` for busybox compatibility
- **Deploy stdin starvation**: Added `</dev/null` to `ssh`/`scp` in
`deploy:install` pipe loop
- **Deploy overwriting UCI**: Noted that `deploy:install` overwrites
`/etc/config/zt-gateway` with repo version, clobbering production
customizations (e.g. real Amsterdam IP 10.11.12.3 vs placeholder
10.99.12.3)

View File

@@ -0,0 +1,124 @@
# ZeroTier Gateway Routing Fixes
## Date: 2026-07-14
## Problem
After switching to the Amsterdam node, traffic from WiBLAN AP clients
(10.11.13.0/24) was not routed through the Amsterdam exit gateway
(10.11.12.3). The table 100 default route pointed correctly to
Amsterdam, but traffic never reached it.
## Root Cause
The `ip rule` directing WIBLAN subnet traffic to policy table 100 was
missing:
```
100: from 10.11.13.0/24 lookup 100
```
Without this rule, WIBLAN client traffic fell through to the main
routing table (priority 32766) which routed via WAN, bypassing the
ZeroTier tunnel entirely.
### Why it was missing
1. **Hotplug script omission**: The hotplug `ifup` case in
`zt-gateway-setup` installed table 100/101 routes and the fwmark
drain rule, but never installed the `from 10.11.13.0/24` source
rule.
2. **No UCI persistence**: `setup-routing` installed the rule at
runtime but its UCI persistence block only wrote route sections,
not a `network rule` section. So netifd couldn't restore it on
boot.
3. **Switch script gap**: `zt-gateway-switch` (`do_force`/`do_graceful`)
assumed the rule already existed and never verified or re-added it.
## Fixes Applied
### 1. Hotplug script (`zt-gateway-setup` heredoc)
Added to the `ifup` case after the table routes:
```sh
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
```
### 2. UCI persistence (`setup-routing`)
Added a `network rule` UCI section so netifd restores the rule on boot:
```sh
uci -q set "network.zt_wiblan_rule=rule"
uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
uci -q set "network.zt_wiblan_rule.priority=100"
```
### 3. Switch script (`zt-gateway-switch`)
Added rule verification to both `do_force()` and `do_graceful()`:
```sh
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
```
## Additional Bugs Fixed
### BRIDGE_PORTS auto-detection failure (busybox)
The `awk` pipeline over `ip -o link show` failed silently on busybox
because busybox awk mishandles `exit` inside compound `if` blocks in
`-F` pipelines. Fixed by:
1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports`
2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'`
3. Validating the result exists: `ip link show "$candidate"`
### Route dev parameter
`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for
default routes, but the ZT interface has no IP after being enslaved
to the bridge. Changed to `dev $BRIDGE` (the bridge device that
holds the 10.11.12.x/23 subnet IP).
### `ip rule replace` busybox incompatibility
Busybox `ip` does not support `ip rule replace`. Replaced all
occurrences with:
```sh
ip rule del ... 2>/dev/null || true
ip rule add ...
```
### Deploy script stdin starvation
The `mise run deploy:install` script used `find | sort | while read`
but `ssh`/`scp` inside the loop consumed stdin from the pipe,
starving the `while read` after the first file. Fixed by adding
`</dev/null` to `ssh` and `scp` commands in the loop.
## Verification Commands
```bash
# Check ip rule exists
ip rule show | grep "lookup 100"
# Expected: 100: from 10.11.13.0/24 lookup 100
# Check table 100 routes
ip route show table 100
# Expected: default via <active_gw_ip> dev br-zt
# Check UCI rule persisted
uci show network | grep zt_wiblan_rule
# Expected: network.zt_wiblan_rule=rule, src, lookup, priority
# Ping exit gateway via bridge
ping -c 2 -I br-zt <active_gw_ip>
```