infra/luci-app-zt-gateway
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: routing ip rule missing after reboot, busybox compat, skill restructure

Browse Source 
Routing fixes (2026-07-14):
- Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case
- Add UCI network rule persistence so netifd restores it on boot
- Verify ip rule exists in zt-gateway-switch do_force/do_graceful
- Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken
  awk-over-ip pipeline (busybox awk mishandles exit in compound if)
- Validate bridge port candidate exists as network interface
- Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP
  when enslaved to bridge, causing 'Nexthop has invalid gateway')
- Replace ip rule replace (GNU-only) with del+add for busybox

Infrastructure:
- Fix deploy:install stdin starvation: ssh/scp consume pipe data in
  find|while loop; add </dev/null to prevent truncation
- Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink
- Add policy routing and busybox gotcha sections to SKILL.md
- Add diagnostics doc for the routing fix session
This commit is contained in:
Malar Invention
2026-07-14 18:01:30 +05:30
parent 10081b33e2
commit 581d625044
9 changed files with 294 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.agents/skills/luci-dev Symbolic link
View File

@@ -0,0 +1 @@
../../skills/luci-dev

39
docs/PROGRESS.md
View File

@@ -153,3 +153,42 @@ After installing `luci-compat`, login and page rendering work correctly, but **f
- ZeroTier requires "Allow Ethernet Bridging" for L2 traffic
- nftables fw4 zones must explicitly include bridge interfaces
- Policy routing table 100 must route via exit gateway, not local IP
## Date: 2026-07-14
## Post-Reboot Routing Fixes
### Problem
After rebooting the production OpenWrt router, WiBLAN clients
(10.11.13.x) could not route traffic through the active exit
gateway (Amsterdam, 10.11.12.3). The `ip rule` directing WIBLAN
traffic to policy table 100 was missing entirely.
### Root Cause
Three gaps in the hotplug/setup-routing/switch pipeline:
1. Hotplug `ifup` case never installed the `from 10.11.13.0/24` ip rule
2. UCI persistence wrote routes but not the `network rule` section
3. `zt-gateway-switch` never verified the rule existed
### Fixes
1. Added ip rule to hotplug heredoc (`zt-gateway-setup`)
2. Added `network.zt_wiblan_rule` UCI section to `setup-routing` persistence
3. Added rule verification to `do_force()`/`do_graceful()` in `zt-gateway-switch`
### Additional fixes during the same session
- **BRIDGE_PORTS auto-detect**: Replaced broken awk-over-ip pipeline
with UCI lookup + `/proc/net/dev` fallback + interface existence
validation (busybox compatibility)
- **Route dev param**: Changed `dev $BRIDGE_PORTS` to `dev $BRIDGE`
in `setup-routing` (ZT interface has no IP when enslaved to bridge)
- **`ip rule replace`**: Replaced GNU-only `ip rule replace` with
`ip rule del` + `ip rule add` for busybox compatibility
- **Deploy stdin starvation**: Added `</dev/null` to `ssh`/`scp` in
`deploy:install` pipe loop
- **Deploy overwriting UCI**: Noted that `deploy:install` overwrites
`/etc/config/zt-gateway` with repo version, clobbering production
customizations (e.g. real Amsterdam IP 10.11.12.3 vs placeholder
10.99.12.3)

124
docs/diagnostics/zerotier-wiblan-missing-iprule-2026-07-14.md Normal file
View File

@@ -0,0 +1,124 @@
# ZeroTier Gateway Routing Fixes
## Date: 2026-07-14
## Problem
After switching to the Amsterdam node, traffic from WiBLAN AP clients
(10.11.13.0/24) was not routed through the Amsterdam exit gateway
(10.11.12.3). The table 100 default route pointed correctly to
Amsterdam, but traffic never reached it.
## Root Cause
The `ip rule` directing WIBLAN subnet traffic to policy table 100 was
missing:
```
100: from 10.11.13.0/24 lookup 100
```
Without this rule, WIBLAN client traffic fell through to the main
routing table (priority 32766) which routed via WAN, bypassing the
ZeroTier tunnel entirely.
### Why it was missing
1. **Hotplug script omission**: The hotplug `ifup` case in
`zt-gateway-setup` installed table 100/101 routes and the fwmark
drain rule, but never installed the `from 10.11.13.0/24` source
rule.
2. **No UCI persistence**: `setup-routing` installed the rule at
runtime but its UCI persistence block only wrote route sections,
not a `network rule` section. So netifd couldn't restore it on
boot.
3. **Switch script gap**: `zt-gateway-switch` (`do_force`/`do_graceful`)
assumed the rule already existed and never verified or re-added it.
## Fixes Applied
### 1. Hotplug script (`zt-gateway-setup` heredoc)
Added to the `ifup` case after the table routes:
```sh
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
```
### 2. UCI persistence (`setup-routing`)
Added a `network rule` UCI section so netifd restores the rule on boot:
```sh
uci -q set "network.zt_wiblan_rule=rule"
uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
uci -q set "network.zt_wiblan_rule.priority=100"
```
### 3. Switch script (`zt-gateway-switch`)
Added rule verification to both `do_force()` and `do_graceful()`:
```sh
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
```
## Additional Bugs Fixed
### BRIDGE_PORTS auto-detection failure (busybox)
The `awk` pipeline over `ip -o link show` failed silently on busybox
because busybox awk mishandles `exit` inside compound `if` blocks in
`-F` pipelines. Fixed by:
1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports`
2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'`
3. Validating the result exists: `ip link show "$candidate"`
### Route dev parameter
`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for
default routes, but the ZT interface has no IP after being enslaved
to the bridge. Changed to `dev $BRIDGE` (the bridge device that
holds the 10.11.12.x/23 subnet IP).
### `ip rule replace` busybox incompatibility
Busybox `ip` does not support `ip rule replace`. Replaced all
occurrences with:
```sh
ip rule del ... 2>/dev/null || true
ip rule add ...
```
### Deploy script stdin starvation
The `mise run deploy:install` script used `find | sort | while read`
but `ssh`/`scp` inside the loop consumed stdin from the pipe,
starving the `while read` after the first file. Fixed by adding
`</dev/null` to `ssh` and `scp` commands in the loop.
## Verification Commands
```bash
# Check ip rule exists
ip rule show | grep "lookup 100"
# Expected: 100: from 10.11.13.0/24 lookup 100
# Check table 100 routes
ip route show table 100
# Expected: default via <active_gw_ip> dev br-zt
# Check UCI rule persisted
uci show network | grep zt_wiblan_rule
# Expected: network.zt_wiblan_rule=rule, src, lookup, priority
# Ping exit gateway via bridge
ping -c 2 -I br-zt <active_gw_ip>
```

4
mise.toml
View File

@@ -98,8 +98,8 @@ find root/ htdocs/ -type f | sort | while IFS= read -r src; do
*) echo "SKIP: $src (unknown prefix)"; continue ;;
esac
destdir="${dest%/*}"
ssh -q "$HOST" mkdir -p "$destdir"
scp -O -q "$src" "$HOST:$dest"
ssh -q "$HOST" mkdir -p "$destdir" </dev/null
scp -O -q "$src" "$HOST:$dest" </dev/null
echo " ${src#./}"
UPLOADED=$((UPLOADED + 1))
done

46
root/usr/sbin/zt-gateway-setup
View File

@@ -52,8 +52,20 @@ BRIDGE="${ZTG_BRIDGE:-br-zt}"
if [ -n "${ZTG_BRIDGE_PORTS:-}" ]; then
BRIDGE_PORTS="$ZTG_BRIDGE_PORTS"
else
BRIDGE_PORTS=$(ip -o link show 2>/dev/null \
| awk -F': ' '/^[0-9]+:/{gsub(/@.*/, "", $2); if ($2 ~ /^zt/ && $2 != "br-zt") print $2; exit}')
# Auto-detect: try UCI first, then /proc/net/dev (busybox-safe),
# then ip link as last resort. Busybox awk mishandles `exit` inside
# compound `if` blocks in `-F` pipelines, so avoid that pattern.
# Validate that any candidate actually exists as a network interface.
BRIDGE_PORTS=$(uci -q get zt-gateway.global.bridge_ports 2>/dev/null || true)
if [ -n "$BRIDGE_PORTS" ] && ! ip link show "$BRIDGE_PORTS" >/dev/null 2>&1; then
BRIDGE_PORTS=""
fi
if [ -z "$BRIDGE_PORTS" ]; then
BRIDGE_PORTS=$(awk -F': ' '/^zt/{print $1; exit}' /proc/net/dev 2>/dev/null || true)
fi
if [ -z "$BRIDGE_PORTS" ]; then
BRIDGE_PORTS=$(ip -br link 2>/dev/null | awk '/^zt/ && !/br-zt/{print $1; exit}')
fi
if [ -z "$BRIDGE_PORTS" ]; then
BRIDGE_PORTS="ztabc0"
fi
@@ -309,11 +321,11 @@ cmd_setup_routing() {
fi
if [ -n "$EXIT_GW" ]; then
log "exit gateway: ${EXIT_GW}"
ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_MAIN"
ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_MAIN"
ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MAIN"
# Table 101 (drain): default via exit gateway
ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_DRAIN"
ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_DRAIN"
else
log "warning: could not determine exit gateway; using WIBLAN_GW"
ip route replace default via "$WIBLAN_GW" dev "$BRIDGE" table "$TABLE_MAIN"
@@ -325,12 +337,12 @@ cmd_setup_routing() {
ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MWAN"
# ip rule: WIBLAN subnet -> main policy table
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \
ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
# ip rule: fwmark 0x100 -> drain table
ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \
ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true
ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
log "routing setup complete"
@@ -362,6 +374,14 @@ cmd_setup_routing() {
uci -q set "network.zt_wiblan_subnet.interface=${BRIDGE}"
uci -q set "network.zt_wiblan_subnet.table=${TABLE_MAIN}"
fi
# Policy rule: WIBLAN subnet → main policy table
if ! uci -q get "network.zt_wiblan_rule" >/dev/null 2>&1; then
uci -q set "network.zt_wiblan_rule=rule"
uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}"
uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}"
uci -q set "network.zt_wiblan_rule.priority=100"
fi
uci commit network
log "routing UCI config committed"
fi
@@ -542,12 +562,16 @@ case "$ACTION" in
# mwan3 return
ip route replace "$ZTG_WIBLAN_CIDR" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MWAN"
# ip rule: WIBLAN subnet → main policy table
ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || true
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
# ip rule for drain fwmark
ip rule del fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || true
ip rule add fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || \
ip rule replace fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \
priority "$ZTG_DRAIN_PRIORITY"
priority "$ZTG_DRAIN_PRIORITY"
;;
ifdown)
logger -t zt-gw-hotplug "ifdown ${ZTG_BRIDGE}: cleaning up"

12
root/usr/sbin/zt-gateway-switch
View File

@@ -199,6 +199,10 @@ do_force() {
preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
# Ensure WIBLAN traffic uses the policy table (may be missing after reboot)
ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
set_host_route "$new_ip"
set_table_default "$new_ip" "$TABLE_MAIN"
ensure_mwan_return
@@ -249,8 +253,8 @@ drain_install_rules() {
old_ip=$1
ip route replace default via "$old_ip" dev "$BRIDGE" table "$TABLE_DRAIN"
mangle_rules_install
ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \
ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true
ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY"
}
drain_cleanup() {
@@ -353,6 +357,10 @@ do_graceful() {
fi
preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}"
# Ensure WIBLAN traffic uses the policy table (may be missing after reboot)
ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true
ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100
printf '%s\n' "$new_ip" >"$DRAIN_NEWFILE"
printf '%s\n' "$old_ip" >"$DRAIN_OLDFILE"

83
luci-dev/SKILL.md → skills/SKILL.md
View File

@@ -179,3 +179,86 @@ Some OpenWrt packages need surgical patches for containerized environments:
let globals = proto({ include: (name, args) => self.render_any(name, args ?? {}) }, scope ?? {});
```
- **`/usr/share/rpcd/ucode/system.uc`**: Only needed if `rpcd-mod-iwinfo` is absent. `luci-mod-admin-full` depends on `rpcd-mod-iwinfo`, so in typical LuCI installs `system.board` is natively available.
## OpenWrt Policy Routing (ZeroTier Exit Gateway)
Policy routing for subnet traffic (e.g. WIBLAN 10.11.13.0/24) requires
three components working together:
### 1. UCI Network Rule (persists across reboot)
```uci
config rule
option src '10.11.13.0/24'
option lookup '100'
option priority '100'
```
This is the **only** component that netifd restores automatically on boot.
Routes in custom tables are also restored, but the `ip rule` that directs
traffic TO those tables must be explicitly defined as a UCI `network rule`.
### 2. Policy Table Routes
Table 100 (main policy) and 101 (drain) must have default routes via the
active exit gateway:
```sh
ip route replace default via <gateway_ip> dev br-zt table 100
ip route replace default via <gateway_ip> dev br-zt table 101
```
**Critical**: Use the bridge device (`br-zt`), not the raw ZeroTier
interface (`ztk4jpk77j`). After the ZT interface is enslaved to the
bridge, it has no IP and `ip route replace default via <gw> dev <zt_if>`
fails with `Nexthop has invalid gateway`.
### 3. Hotplug Script
Re-applies routes when the bridge comes up (e.g. after ZeroTier restart).
Must install BOTH the table routes AND the ip rule:
```sh
# Table routes
ip route replace default via "$active_ip" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MAIN"
# Ip rule (commonly forgotten!)
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null
ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
```
### Verification
```bash
# Rule exists?
ip rule show | grep "lookup 100"
# Table has correct default?
ip route show table 100
# Gateway reachable via bridge?
ping -c 2 -I br-zt <gateway_ip>
```
## Busybox / OpenWrt Shell Gotchas
1. **`ip rule replace` does not exist** in busybox `ip`. Use
`ip rule del ... 2>/dev/null || true; ip rule add ...` instead.
2. **`awk exit` in compound if blocks**: Busybox awk mishandles `exit`
inside `if (...) { ...; exit }` when used with `-F` field separator
in a pipeline. Workaround: use `/proc/net/dev` as input instead of
piping from `ip -o link show`, or use multi-line awk programs.
3. **Shell pipe + ssh stdin starvation**: `ssh` and `scp` consume stdin.
In `find | sort | while read; do ssh ...; done` loops, add `</dev/null`
to `ssh`/`scp` commands to prevent them from eating the pipe data.
4. **UCI `network rule` vs runtime `ip rule add`**: `ip rule add` only
persists in the running kernel. For boot survival, write a UCI
`network rule` section so netifd applies it. But note: netifd may
not handle `ip rule replace` idempotency — always `del`+`add`.
5. **Bridge ports auto-detection**: Don't rely on `ip -o link show | awk`
for interface detection on busybox. Prefer reading `/proc/net/dev` or
`/sys/class/net/` which are simpler and more portable. Always validate
the detected interface actually exists with `ip link show <name>`.

0
luci-dev/TODO.md → skills/TODO.md
View File

0
luci-dev/references/openwrt-docker-build.md → skills/references/openwrt-docker-build.md
View File