Malar Invention
581d625044
Routing fixes (2026-07-14): - Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case - Add UCI network rule persistence so netifd restores it on boot - Verify ip rule exists in zt-gateway-switch do_force/do_graceful - Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken awk-over-ip pipeline (busybox awk mishandles exit in compound if) - Validate bridge port candidate exists as network interface - Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP when enslaved to bridge, causing 'Nexthop has invalid gateway') - Replace ip rule replace (GNU-only) with del+add for busybox Infrastructure: - Fix deploy:install stdin starvation: ssh/scp consume pipe data in find|while loop; add </dev/null to prevent truncation - Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink - Add policy routing and busybox gotcha sections to SKILL.md - Add diagnostics doc for the routing fix session
125 lines
3.6 KiB
Markdown
125 lines
3.6 KiB
Markdown
# ZeroTier Gateway Routing Fixes
|
|
|
|
## Date: 2026-07-14
|
|
|
|
## Problem
|
|
|
|
After switching to the Amsterdam node, traffic from WiBLAN AP clients
|
|
(10.11.13.0/24) was not routed through the Amsterdam exit gateway
|
|
(10.11.12.3). The table 100 default route pointed correctly to
|
|
Amsterdam, but traffic never reached it.
|
|
|
|
## Root Cause
|
|
|
|
The `ip rule` directing WIBLAN subnet traffic to policy table 100 was
|
|
missing:
|
|
|
|
```
|
|
100: from 10.11.13.0/24 lookup 100
|
|
```
|
|
|
|
Without this rule, WIBLAN client traffic fell through to the main
|
|
routing table (priority 32766) which routed via WAN, bypassing the
|
|
ZeroTier tunnel entirely.
|
|
|
|
### Why it was missing
|
|
|
|
1. **Hotplug script omission**: The hotplug `ifup` case in
|
|
`zt-gateway-setup` installed table 100/101 routes and the fwmark
|
|
drain rule, but never installed the `from 10.11.13.0/24` source
|
|
rule.
|
|
|
|
2. **No UCI persistence**: `setup-routing` installed the rule at
|
|
runtime but its UCI persistence block only wrote route sections,
|
|
not a `network rule` section. So netifd couldn't restore it on
|
|
boot.
|
|
|
|
3. **Switch script gap**: `zt-gateway-switch` (`do_force`/`do_graceful`)
|
|
assumed the rule already existed and never verified or re-added it.
|
|
|
|
## Fixes Applied
|
|
|
|
### 1. Hotplug script (`zt-gateway-setup` heredoc)
|
|
|
|
Added to the `ifup` case after the table routes:
|
|
|
|
```sh
|
|
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
|
|
ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
|
|
```
|
|
|
|
### 2. UCI persistence (`setup-routing`)
|
|
|
|
Added a `network rule` UCI section so netifd restores the rule on boot:
|
|
|
|
```sh
|
|
uci -q set "network.zt_wiblan_rule=rule"
|
|
uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
|
|
uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
|
|
uci -q set "network.zt_wiblan_rule.priority=100"
|
|
```
|
|
|
|
### 3. Switch script (`zt-gateway-switch`)
|
|
|
|
Added rule verification to both `do_force()` and `do_graceful()`:
|
|
|
|
```sh
|
|
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
|
|
ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
|
|
```
|
|
|
|
## Additional Bugs Fixed
|
|
|
|
### BRIDGE_PORTS auto-detection failure (busybox)
|
|
|
|
The `awk` pipeline over `ip -o link show` failed silently on busybox
|
|
because busybox awk mishandles `exit` inside compound `if` blocks in
|
|
`-F` pipelines. Fixed by:
|
|
|
|
1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports`
|
|
2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'`
|
|
3. Validating the result exists: `ip link show "$candidate"`
|
|
|
|
### Route dev parameter
|
|
|
|
`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for
|
|
default routes, but the ZT interface has no IP after being enslaved
|
|
to the bridge. Changed to `dev $BRIDGE` (the bridge device that
|
|
holds the 10.11.12.x/23 subnet IP).
|
|
|
|
### `ip rule replace` busybox incompatibility
|
|
|
|
Busybox `ip` does not support `ip rule replace`. Replaced all
|
|
occurrences with:
|
|
|
|
```sh
|
|
ip rule del ... 2>/dev/null || true
|
|
ip rule add ...
|
|
```
|
|
|
|
### Deploy script stdin starvation
|
|
|
|
The `mise run deploy:install` script used `find | sort | while read`
|
|
but `ssh`/`scp` inside the loop consumed stdin from the pipe,
|
|
starving the `while read` after the first file. Fixed by adding
|
|
`</dev/null` to `ssh` and `scp` commands in the loop.
|
|
|
|
## Verification Commands
|
|
|
|
```bash
|
|
# Check ip rule exists
|
|
ip rule show | grep "lookup 100"
|
|
# Expected: 100: from 10.11.13.0/24 lookup 100
|
|
|
|
# Check table 100 routes
|
|
ip route show table 100
|
|
# Expected: default via <active_gw_ip> dev br-zt
|
|
|
|
# Check UCI rule persisted
|
|
uci show network | grep zt_wiblan_rule
|
|
# Expected: network.zt_wiblan_rule=rule, src, lookup, priority
|
|
|
|
# Ping exit gateway via bridge
|
|
ping -c 2 -I br-zt <active_gw_ip>
|
|
```
|