diff --git a/.agents/skills/luci-dev b/.agents/skills/luci-dev new file mode 120000 index 0000000..b1b2afd --- /dev/null +++ b/.agents/skills/luci-dev @@ -0,0 +1 @@ +../../skills/luci-dev \ No newline at end of file diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 53f6977..2db28c7 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -153,3 +153,42 @@ After installing `luci-compat`, login and page rendering work correctly, but **f - ZeroTier requires "Allow Ethernet Bridging" for L2 traffic - nftables fw4 zones must explicitly include bridge interfaces - Policy routing table 100 must route via exit gateway, not local IP +## Date: 2026-07-14 + +## Post-Reboot Routing Fixes + +### Problem + +After rebooting the production OpenWrt router, WiBLAN clients +(10.11.13.x) could not route traffic through the active exit +gateway (Amsterdam, 10.11.12.3). The `ip rule` directing WIBLAN +traffic to policy table 100 was missing entirely. + +### Root Cause + +Three gaps in the hotplug/setup-routing/switch pipeline: +1. Hotplug `ifup` case never installed the `from 10.11.13.0/24` ip rule +2. UCI persistence wrote routes but not the `network rule` section +3. `zt-gateway-switch` never verified the rule existed + +### Fixes + +1. Added ip rule to hotplug heredoc (`zt-gateway-setup`) +2. Added `network.zt_wiblan_rule` UCI section to `setup-routing` persistence +3. Added rule verification to `do_force()`/`do_graceful()` in `zt-gateway-switch` + +### Additional fixes during the same session + +- **BRIDGE_PORTS auto-detect**: Replaced broken awk-over-ip pipeline + with UCI lookup + `/proc/net/dev` fallback + interface existence + validation (busybox compatibility) +- **Route dev param**: Changed `dev $BRIDGE_PORTS` to `dev $BRIDGE` + in `setup-routing` (ZT interface has no IP when enslaved to bridge) +- **`ip rule replace`**: Replaced GNU-only `ip rule replace` with + `ip rule del` + `ip rule add` for busybox compatibility +- **Deploy stdin starvation**: Added `/dev/null || \ + ip rule replace from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 +``` + +### 2. UCI persistence (`setup-routing`) + +Added a `network rule` UCI section so netifd restores the rule on boot: + +```sh +uci -q set "network.zt_wiblan_rule=rule" +uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}" +uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}" +uci -q set "network.zt_wiblan_rule.priority=100" +``` + +### 3. Switch script (`zt-gateway-switch`) + +Added rule verification to both `do_force()` and `do_graceful()`: + +```sh +ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \ + ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 +``` + +## Additional Bugs Fixed + +### BRIDGE_PORTS auto-detection failure (busybox) + +The `awk` pipeline over `ip -o link show` failed silently on busybox +because busybox awk mishandles `exit` inside compound `if` blocks in +`-F` pipelines. Fixed by: + +1. Trying UCI config first: `uci -q get zt-gateway.global.bridge_ports` +2. Falling back to `/proc/net/dev`: `awk -F': ' '/^zt/{print $1; exit}'` +3. Validating the result exists: `ip link show "$candidate"` + +### Route dev parameter + +`setup-routing` used `dev $BRIDGE_PORTS` (raw ZT interface) for +default routes, but the ZT interface has no IP after being enslaved +to the bridge. Changed to `dev $BRIDGE` (the bridge device that +holds the 10.11.12.x/23 subnet IP). + +### `ip rule replace` busybox incompatibility + +Busybox `ip` does not support `ip rule replace`. Replaced all +occurrences with: + +```sh +ip rule del ... 2>/dev/null || true +ip rule add ... +``` + +### Deploy script stdin starvation + +The `mise run deploy:install` script used `find | sort | while read` +but `ssh`/`scp` inside the loop consumed stdin from the pipe, +starving the `while read` after the first file. Fixed by adding +` dev br-zt + +# Check UCI rule persisted +uci show network | grep zt_wiblan_rule +# Expected: network.zt_wiblan_rule=rule, src, lookup, priority + +# Ping exit gateway via bridge +ping -c 2 -I br-zt +``` diff --git a/mise.toml b/mise.toml index 0126a62..d46c176 100644 --- a/mise.toml +++ b/mise.toml @@ -98,8 +98,8 @@ find root/ htdocs/ -type f | sort | while IFS= read -r src; do *) echo "SKIP: $src (unknown prefix)"; continue ;; esac destdir="${dest%/*}" - ssh -q "$HOST" mkdir -p "$destdir" - scp -O -q "$src" "$HOST:$dest" + ssh -q "$HOST" mkdir -p "$destdir" /dev/null \ - | awk -F': ' '/^[0-9]+:/{gsub(/@.*/, "", $2); if ($2 ~ /^zt/ && $2 != "br-zt") print $2; exit}') + # Auto-detect: try UCI first, then /proc/net/dev (busybox-safe), + # then ip link as last resort. Busybox awk mishandles `exit` inside + # compound `if` blocks in `-F` pipelines, so avoid that pattern. + # Validate that any candidate actually exists as a network interface. + BRIDGE_PORTS=$(uci -q get zt-gateway.global.bridge_ports 2>/dev/null || true) + if [ -n "$BRIDGE_PORTS" ] && ! ip link show "$BRIDGE_PORTS" >/dev/null 2>&1; then + BRIDGE_PORTS="" + fi + if [ -z "$BRIDGE_PORTS" ]; then + BRIDGE_PORTS=$(awk -F': ' '/^zt/{print $1; exit}' /proc/net/dev 2>/dev/null || true) + fi + if [ -z "$BRIDGE_PORTS" ]; then + BRIDGE_PORTS=$(ip -br link 2>/dev/null | awk '/^zt/ && !/br-zt/{print $1; exit}') + fi if [ -z "$BRIDGE_PORTS" ]; then BRIDGE_PORTS="ztabc0" fi @@ -309,11 +321,11 @@ cmd_setup_routing() { fi if [ -n "$EXIT_GW" ]; then log "exit gateway: ${EXIT_GW}" - ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_MAIN" + ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_MAIN" ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MAIN" # Table 101 (drain): default via exit gateway - ip route replace default via "$EXIT_GW" dev "$BRIDGE_PORTS" table "$TABLE_DRAIN" + ip route replace default via "$EXIT_GW" dev "$BRIDGE" table "$TABLE_DRAIN" else log "warning: could not determine exit gateway; using WIBLAN_GW" ip route replace default via "$WIBLAN_GW" dev "$BRIDGE" table "$TABLE_MAIN" @@ -325,12 +337,12 @@ cmd_setup_routing() { ip route replace "$WIBLAN_CIDR" dev "$BRIDGE" table "$TABLE_MWAN" # ip rule: WIBLAN subnet -> main policy table - ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || \ - ip rule replace from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 + ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true + ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 # ip rule: fwmark 0x100 -> drain table - ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \ - ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" + ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true + ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" log "routing setup complete" @@ -362,6 +374,14 @@ cmd_setup_routing() { uci -q set "network.zt_wiblan_subnet.interface=${BRIDGE}" uci -q set "network.zt_wiblan_subnet.table=${TABLE_MAIN}" fi + # Policy rule: WIBLAN subnet → main policy table + if ! uci -q get "network.zt_wiblan_rule" >/dev/null 2>&1; then + uci -q set "network.zt_wiblan_rule=rule" + uci -q set "network.zt_wiblan_rule.src=${WIBLAN_CIDR}" + uci -q set "network.zt_wiblan_rule.lookup=${TABLE_MAIN}" + uci -q set "network.zt_wiblan_rule.priority=100" + fi + uci commit network log "routing UCI config committed" fi @@ -542,12 +562,16 @@ case "$ACTION" in # mwan3 return ip route replace "$ZTG_WIBLAN_CIDR" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MWAN" + # ip rule: WIBLAN subnet → main policy table + ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || true + ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 + # ip rule for drain fwmark + ip rule del fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \ + priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || true ip rule add fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \ - priority "$ZTG_DRAIN_PRIORITY" 2>/dev/null || \ - ip rule replace fwmark "$ZTG_FWMARK" table "$ZTG_TABLE_DRAIN" \ - priority "$ZTG_DRAIN_PRIORITY" + priority "$ZTG_DRAIN_PRIORITY" ;; ifdown) logger -t zt-gw-hotplug "ifdown ${ZTG_BRIDGE}: cleaning up" diff --git a/root/usr/sbin/zt-gateway-switch b/root/usr/sbin/zt-gateway-switch index b5e753f..10d8095 100755 --- a/root/usr/sbin/zt-gateway-switch +++ b/root/usr/sbin/zt-gateway-switch @@ -199,6 +199,10 @@ do_force() { preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}" + # Ensure WIBLAN traffic uses the policy table (may be missing after reboot) + ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true + ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 + set_host_route "$new_ip" set_table_default "$new_ip" "$TABLE_MAIN" ensure_mwan_return @@ -249,8 +253,8 @@ drain_install_rules() { old_ip=$1 ip route replace default via "$old_ip" dev "$BRIDGE" table "$TABLE_DRAIN" mangle_rules_install - ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || \ - ip rule replace fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" + ip rule del fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" 2>/dev/null || true + ip rule add fwmark "$FWMARK" table "$TABLE_DRAIN" priority "$DRAIN_PRIORITY" } drain_cleanup() { @@ -353,6 +357,10 @@ do_graceful() { fi preflight_ping "$new_ip" || die 2 "gateway ${new_ip} is unreachable over ${PING_IFACE}" + # Ensure WIBLAN traffic uses the policy table (may be missing after reboot) + ip rule del from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 2>/dev/null || true + ip rule add from "$WIBLAN_CIDR" table "$TABLE_MAIN" priority 100 + printf '%s\n' "$new_ip" >"$DRAIN_NEWFILE" printf '%s\n' "$old_ip" >"$DRAIN_OLDFILE" diff --git a/luci-dev/SKILL.md b/skills/SKILL.md similarity index 74% rename from luci-dev/SKILL.md rename to skills/SKILL.md index 02ed9da..585d502 100644 --- a/luci-dev/SKILL.md +++ b/skills/SKILL.md @@ -179,3 +179,86 @@ Some OpenWrt packages need surgical patches for containerized environments: let globals = proto({ include: (name, args) => self.render_any(name, args ?? {}) }, scope ?? {}); ``` - **`/usr/share/rpcd/ucode/system.uc`**: Only needed if `rpcd-mod-iwinfo` is absent. `luci-mod-admin-full` depends on `rpcd-mod-iwinfo`, so in typical LuCI installs `system.board` is natively available. + + +## OpenWrt Policy Routing (ZeroTier Exit Gateway) + +Policy routing for subnet traffic (e.g. WIBLAN 10.11.13.0/24) requires +three components working together: + +### 1. UCI Network Rule (persists across reboot) + +```uci +config rule + option src '10.11.13.0/24' + option lookup '100' + option priority '100' +``` + +This is the **only** component that netifd restores automatically on boot. +Routes in custom tables are also restored, but the `ip rule` that directs +traffic TO those tables must be explicitly defined as a UCI `network rule`. + +### 2. Policy Table Routes + +Table 100 (main policy) and 101 (drain) must have default routes via the +active exit gateway: + +```sh +ip route replace default via dev br-zt table 100 +ip route replace default via dev br-zt table 101 +``` + +**Critical**: Use the bridge device (`br-zt`), not the raw ZeroTier +interface (`ztk4jpk77j`). After the ZT interface is enslaved to the +bridge, it has no IP and `ip route replace default via dev ` +fails with `Nexthop has invalid gateway`. + +### 3. Hotplug Script + +Re-applies routes when the bridge comes up (e.g. after ZeroTier restart). +Must install BOTH the table routes AND the ip rule: + +```sh +# Table routes +ip route replace default via "$active_ip" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MAIN" +# Ip rule (commonly forgotten!) +ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \ + ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null +ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 +``` + +### Verification + +```bash +# Rule exists? +ip rule show | grep "lookup 100" +# Table has correct default? +ip route show table 100 +# Gateway reachable via bridge? +ping -c 2 -I br-zt +``` + +## Busybox / OpenWrt Shell Gotchas + +1. **`ip rule replace` does not exist** in busybox `ip`. Use + `ip rule del ... 2>/dev/null || true; ip rule add ...` instead. + +2. **`awk exit` in compound if blocks**: Busybox awk mishandles `exit` + inside `if (...) { ...; exit }` when used with `-F` field separator + in a pipeline. Workaround: use `/proc/net/dev` as input instead of + piping from `ip -o link show`, or use multi-line awk programs. + +3. **Shell pipe + ssh stdin starvation**: `ssh` and `scp` consume stdin. + In `find | sort | while read; do ssh ...; done` loops, add ``. \ No newline at end of file diff --git a/luci-dev/TODO.md b/skills/TODO.md similarity index 100% rename from luci-dev/TODO.md rename to skills/TODO.md diff --git a/luci-dev/references/openwrt-docker-build.md b/skills/references/openwrt-docker-build.md similarity index 100% rename from luci-dev/references/openwrt-docker-build.md rename to skills/references/openwrt-docker-build.md