fix: routing ip rule missing after reboot, busybox compat, skill restructure
Routing fixes (2026-07-14): - Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case - Add UCI network rule persistence so netifd restores it on boot - Verify ip rule exists in zt-gateway-switch do_force/do_graceful - Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken awk-over-ip pipeline (busybox awk mishandles exit in compound if) - Validate bridge port candidate exists as network interface - Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP when enslaved to bridge, causing 'Nexthop has invalid gateway') - Replace ip rule replace (GNU-only) with del+add for busybox Infrastructure: - Fix deploy:install stdin starvation: ssh/scp consume pipe data in find|while loop; add </dev/null to prevent truncation - Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink - Add policy routing and busybox gotcha sections to SKILL.md - Add diagnostics doc for the routing fix session
This commit is contained in:
@@ -153,3 +153,42 @@ After installing `luci-compat`, login and page rendering work correctly, but **f
|
||||
- ZeroTier requires "Allow Ethernet Bridging" for L2 traffic
|
||||
- nftables fw4 zones must explicitly include bridge interfaces
|
||||
- Policy routing table 100 must route via exit gateway, not local IP
|
||||
## Date: 2026-07-14
|
||||
|
||||
## Post-Reboot Routing Fixes
|
||||
|
||||
### Problem
|
||||
|
||||
After rebooting the production OpenWrt router, WiBLAN clients
|
||||
(10.11.13.x) could not route traffic through the active exit
|
||||
gateway (Amsterdam, 10.11.12.3). The `ip rule` directing WIBLAN
|
||||
traffic to policy table 100 was missing entirely.
|
||||
|
||||
### Root Cause
|
||||
|
||||
Three gaps in the hotplug/setup-routing/switch pipeline:
|
||||
1. Hotplug `ifup` case never installed the `from 10.11.13.0/24` ip rule
|
||||
2. UCI persistence wrote routes but not the `network rule` section
|
||||
3. `zt-gateway-switch` never verified the rule existed
|
||||
|
||||
### Fixes
|
||||
|
||||
1. Added ip rule to hotplug heredoc (`zt-gateway-setup`)
|
||||
2. Added `network.zt_wiblan_rule` UCI section to `setup-routing` persistence
|
||||
3. Added rule verification to `do_force()`/`do_graceful()` in `zt-gateway-switch`
|
||||
|
||||
### Additional fixes during the same session
|
||||
|
||||
- **BRIDGE_PORTS auto-detect**: Replaced broken awk-over-ip pipeline
|
||||
with UCI lookup + `/proc/net/dev` fallback + interface existence
|
||||
validation (busybox compatibility)
|
||||
- **Route dev param**: Changed `dev $BRIDGE_PORTS` to `dev $BRIDGE`
|
||||
in `setup-routing` (ZT interface has no IP when enslaved to bridge)
|
||||
- **`ip rule replace`**: Replaced GNU-only `ip rule replace` with
|
||||
`ip rule del` + `ip rule add` for busybox compatibility
|
||||
- **Deploy stdin starvation**: Added `</dev/null` to `ssh`/`scp` in
|
||||
`deploy:install` pipe loop
|
||||
- **Deploy overwriting UCI**: Noted that `deploy:install` overwrites
|
||||
`/etc/config/zt-gateway` with repo version, clobbering production
|
||||
customizations (e.g. real Amsterdam IP 10.11.12.3 vs placeholder
|
||||
10.99.12.3)
|
||||
|
||||
124
docs/diagnostics/zerotier-wiblan-missing-iprule-2026-07-14.md
Normal file
124
docs/diagnostics/zerotier-wiblan-missing-iprule-2026-07-14.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# ZeroTier Gateway Routing Fixes
|
||||
|
||||
## Date: 2026-07-14
|
||||
|
||||
## Problem
|
||||
|
||||
After switching to the Amsterdam node, traffic from WiBLAN AP clients
|
||||
(10.11.13.0/24) was not routed through the Amsterdam exit gateway
|
||||
(10.11.12.3). The table 100 default route pointed correctly to
|
||||
Amsterdam, but traffic never reached it.
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `ip rule` directing WIBLAN subnet traffic to policy table 100 was
|
||||
missing:
|
||||
|
||||
```
|
||||
100: from 10.11.13.0/24 lookup 100
|
||||
```
|
||||
|
||||
Without this rule, WIBLAN client traffic fell through to the main
|
||||
routing table (priority 32766) which routed via WAN, bypassing the
|
||||
ZeroTier tunnel entirely.
|
||||
|
||||
### Why it was missing
|
||||
|
||||
1. **Hotplug script omission**: The hotplug `ifup` case in
|
||||
`zt-gateway-setup` installed table 100/101 routes and the fwmark
|
||||
drain rule, but never installed the `from 10.11.13.0/24` source
|
||||
rule.
|
||||
|
||||
2. **No UCI persistence**: `setup-routing` installed the rule at
|
||||
runtime but its UCI persistence block only wrote route sections,
|
||||
not a `network rule` section. So netifd couldn't restore it on
|
||||
boot.
|
||||
|
||||
3. **Switch script gap**: `zt-gateway-switch` (`do_force`/`do_graceful`)
|
||||
assumed the rule already existed and never verified or re-added it.
|
||||
|
||||
## Fixes Applied
|
||||
|
||||
### 1. Hotplug script (`zt-gateway-setup` heredoc)
|
||||
|
||||
Added to the `ifup` case after the table routes:
|
||||
|
||||
```sh
|
||||
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
|
||||
ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
|
||||
```
|
||||
|
||||
### 2. UCI persistence (`setup-routing`)
|
||||
|
||||
Added a `network rule` UCI section so netifd restores the rule on boot:
|
||||
|
||||
```sh
|
||||
uci -q set "network.zt_wiblan_rule=rule"
|
||||
uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
|
||||
uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
|
||||
uci -q set "network.zt_wiblan_rule.priority=100"
|
||||
```
|
||||
|
||||
### 3. Switch script (`zt-gateway-switch`)
|
||||
|
||||
Added rule verification to both `do_force()` and `do_graceful()`:
|
||||
|
||||
```sh
|
||||
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
|
||||
ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
|
||||
```
|
||||
|
||||
## Additional Bugs Fixed
|
||||
|
||||
### BRIDGE_PORTS auto-detection failure (busybox)
|
||||
|
||||
The `awk` pipeline over `ip -o link show` failed silently on busybox
|
||||
because busybox awk mishandles `exit` inside compound `if` blocks in
|
||||
`-F` pipelines. Fixed by:
|
||||
|
||||
1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports`
|
||||
2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'`
|
||||
3. Validating the result exists: `ip link show "$candidate"`
|
||||
|
||||
### Route dev parameter
|
||||
|
||||
`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for
|
||||
default routes, but the ZT interface has no IP after being enslaved
|
||||
to the bridge. Changed to `dev $BRIDGE` (the bridge device that
|
||||
holds the 10.11.12.x/23 subnet IP).
|
||||
|
||||
### `ip rule replace` busybox incompatibility
|
||||
|
||||
Busybox `ip` does not support `ip rule replace`. Replaced all
|
||||
occurrences with:
|
||||
|
||||
```sh
|
||||
ip rule del ... 2>/dev/null || true
|
||||
ip rule add ...
|
||||
```
|
||||
|
||||
### Deploy script stdin starvation
|
||||
|
||||
The `mise run deploy:install` script used `find | sort | while read`
|
||||
but `ssh`/`scp` inside the loop consumed stdin from the pipe,
|
||||
starving the `while read` after the first file. Fixed by adding
|
||||
`</dev/null` to `ssh` and `scp` commands in the loop.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Check ip rule exists
|
||||
ip rule show | grep "lookup 100"
|
||||
# Expected: 100: from 10.11.13.0/24 lookup 100
|
||||
|
||||
# Check table 100 routes
|
||||
ip route show table 100
|
||||
# Expected: default via <active_gw_ip> dev br-zt
|
||||
|
||||
# Check UCI rule persisted
|
||||
uci show network | grep zt_wiblan_rule
|
||||
# Expected: network.zt_wiblan_rule=rule, src, lookup, priority
|
||||
|
||||
# Ping exit gateway via bridge
|
||||
ping -c 2 -I br-zt <active_gw_ip>
|
||||
```
|
||||
Reference in New Issue
Block a user