fix: routing ip rule missing after reboot, busybox compat, skill restructure

Routing fixes (2026-07-14): - Add missing ip rule 'from 10.11.13.0/24 lookup 100' to hotplug ifup case - Add UCI network rule persistence so netifd restores it on boot - Verify ip rule exists in zt-gateway-switch do_force/do_graceful - Fix BRIDGE_PORTS auto-detect: use /proc/net/dev instead of broken awk-over-ip pipeline (busybox awk mishandles exit in compound if) - Validate bridge port candidate exists as network interface - Fix setup-routing: use dev br-zt not dev ztX (ZT iface has no IP when enslaved to bridge, causing 'Nexthop has invalid gateway') - Replace ip rule replace (GNU-only) with del+add for busybox Infrastructure: - Fix deploy:install stdin starvation: ssh/scp consume pipe data in find|while loop; add </dev/null to prevent truncation - Move luci-dev skill from root/ to skills/ with .agents/skills/ symlink - Add policy routing and busybox gotcha sections to SKILL.md - Add diagnostics doc for the routing fix session
2026-07-14 18:01:30 +05:30
parent 10081b33e2
commit 581d625044
9 changed files with 294 additions and 15 deletions
--- a/skills/SKILL.md
+++ b/skills/SKILL.md
@@ -0,0 +1,264 @@
+---
+name: luci-dev
+description: OpenWrt LuCI application development, Docker containerization, and Playwright E2E testing. Use when working on luci-app-* packages, rpcd-mod-ucode backends, LuCI JavaScript views, or building reproducible E2E test stacks for OpenWrt web UIs. Covers apk 3.0 package management, ubus RPC patterns, Playwright serial test strategies, and Docker build workarounds for OpenWrt rootfs containers.
+---
+
+# LuCI Dev
+
+## Overview
+
+This skill covers the full workflow for developing LuCI (Lua/UCI) applications on modern OpenWrt SNAPSHOT / 25.12+:
+- Writing rpcd-mod-ucode backends (`/usr/share/rpcd/ucode/*.uc`)
+- Writing LuCI JavaScript views (`htdocs/luci-static/resources/view/**/*.js`)
+- Dockerizing the OpenWrt rootfs for reproducible E2E testing
+- Playwright E2E tests that run serially against a shared router state container
+
+## OpenWrt apk 3.0 Docker Build
+
+OpenWrt SNAPSHOT / 25.12+ uses Alpine's `apk` package manager (v3). Key gotchas for Docker builds:
+
+1. **Repo URLs must point to `packages.adb` directly** — not directories:
+   ```
+   https://downloads.openwrt.org/snapshots/packages/x86_64/base/packages.adb
+   ```
+   Use `snapshots/` URLs for the latest `openwrt/rootfs:latest` base image; use `releases/25.12.4/` only if you are pinned to that release.
+2. **`--allow-untrusted`** is required because the base rootfs lacks OpenWrt signing keys
+3. **BuildKit seccomp blocks OpenWrt's `wget`/`uclient-fetch`** — `RUN --security=insecure` in a Dockerfile does **not** help when the Docker daemon is backed by Podman, because Podman applies its own seccomp profile at a lower level than BuildKit can override. The pragmatic fix is:
+   - **Use `podman build`** (shares image storage with Docker on Podman-backed systems):
+     ```bash
+     podman build --security-opt seccomp=unconfined -t zt-gateway-luci:dev -f Dockerfile.openwrt .
+     ```
+   - Do NOT use `docker buildx` or `docker compose --build` for the `openwrt-luci` image in Podman environments.
+4. **Create `/var/lock` and `/var/run`** before `apk add` so post-install scripts don't fail (they try to create procd lockfiles)
+5. **Kernel modules (`kmod-*`) and `openwrt-kernel` don't exist as apk packages** — skip them in Docker; the container runs against the host kernel anyway
+6. **`uhttpd` and `luci-theme-bootstrap` must be explicitly installed** — the base `openwrt/rootfs` image only contains `ubusd`, not `rpcd`, `uhttpd`, LuCI, or any theme. Without a theme, LuCI fails to render with "Unable to render any theme header template".
+
+See `references/openwrt-docker-build.md` for the full Dockerfile template and build command.
+
+## rpcd-mod-ucode Backend
+
+File: `/usr/share/rpcd/ucode/*.uc`
+
+Format:
+```javascript
+'use strict';
+
+function helper() { /* ... */ }
+
+return {
+    'namespace-name': {
+        methodName: {
+            args: { param: 'string' },
+            call: function(req) {
+                const value = req.args?.param;
+                return { result: value };
+            }
+        }
+    }
+};
+```
+
+Critical rules:
+- **Parameters live in `req.args.param`**, not `msg.param` or positional args
+- **Use `'use strict';`** at the top
+- **Function declarations are NOT hoisted** in ucode strict mode — define helpers BEFORE they are called
+- **`return` the response object directly**; don't use `ubus.reply()`
+- Use `match(string, /regex/)` instead of `~` operator
+- Use `cursor.foreach('config', 'section-type', function(s) { ... })` instead of `uci.sections()`
+- Use `time()` for timestamps; `getpid()` doesn't exist in rpcd-ucode context
+- **`system_output()` style functions** (redirect to tmp file + read + unlink) work in ucode but are synchronous and blocking — set generous client timeouts
+
+## LuCI JavaScript Frontend
+
+File: `htdocs/luci-static/resources/view/<app>/<view>.js`
+
+LuCI uses its own module system, not ES modules:
+```javascript
+'use strict';
+'require view';
+'require rpc';
+'require ui';
+'require dom';
+'require poll';
+
+function ubusStatus() {
+    return rpc.declare({
+        object: 'namespace',
+        method: 'status',
+        params: []
+    })();
+}
+
+return view.extend({
+    load() { /* ... */ },
+    render(data) { /* ... */ }
+});
+```
+
+Critical rules:
+- **Use `'require view';` string directives** at the top (NOT `import`)
+- **`Poll.add(fn, interval)`** is the correct API — `Poll.create()` does not exist
+- **Boolean HTML attributes need `|| null`**:
+  - `disabled: isActive || null` (NOT `disabled: isActive`, because `disabled="false"` still disables the element)
+  - `checked: isSelected || null`
+- Use `E('tag', { attrs }, children)` for DOM construction
+- Use `_(...)` for internationalization strings
+- **Health dots render initially as DOWN** (`renderHealthDot(false, null)`), then `refreshHealth()` updates them asynchronously after the first poll cycle
+
+## Docker/Compose Stack for E2E
+
+The `openwrt-luci` service in `docker-compose.yml`:
+- Uses `privileged: true` and `NET_ADMIN`/`SYS_ADMIN` caps
+- Sits on a custom bridge network (`zt-exit-net`) so it can ping mock gateways
+- Entrypoint must start `ubusd` first, then `rpcd`, then `uhttpd -f -p 80 -h /www -u /ubus -a`
+- Root password should be empty (`root::0:0:99999:7:::` in `/etc/shadow`) for Playwright login
+- First page load after container start takes ~15-20s due to ucode template compilation; subsequent loads are fast
+
+Mock gateway containers should be simple `archlinux` containers on the same bridge that `sleep infinity` — the kernel responds to ping for their assigned IPs. They need no special configuration apart from being on the same Docker network.
+
+**Volume mounts in compose** are the pragmatic development path: overlay fixed repo files (`zt-gateway.uc`, `overview.js`, `zt-gateway` config, `entrypoint.sh`) onto the running container without rebuilding the image.
+
+## Playwright E2E Testing
+
+Config must enforce serial execution because tests mutate shared router state:
+```typescript
+// playwright.config.ts
+export default defineConfig({
+    fullyParallel: false,
+    workers: 1,
+});
+```
+
+Test patterns:
+- **Login**: Submit LuCI login form with empty password; dismiss "No password set" notification if present
+  - Username field: `#luci_username`
+  - Password field: `#luci_password`
+  - Submit button: `.cbi-button-positive`
+  - After login, URL contains `/cgi-bin/luci/admin/`
+- **Overview**: Navigate to `/cgi-bin/luci/admin/services/zt-gateway`, verify `#zt-gateway-root`, gateway table, active gateway highlight
+  - Table rows: `.zt-gateways tbody tr.zt-gateway-row`
+  - Row identification: `.zt-gateway-row[data-region="amsterdam"]`
+  - Active marker text: "active" inside the row
+  - Active row radio is `disabled`; non-active radios are `enabled`
+- **Switch**: Select radio for non-active gateway, click "Switch to selected", poll until active gateway changes
+  - Switch RPC takes **12-30s** because `zt-gateway-switch` preflight_ping retries without `-I` interface binding on failure
+  - Set generous timeouts (45s+) for switch operations
+- **Drain**: Select graceful mode, verify drain panel appears, test cancel drain
+  - Mode dropdown: `.zt-mode-select`
+  - When graceful is selected, `.zt-drain-timeout-row` becomes visible
+  - After graceful switch, `.zt-drain-panel` appears with "Graceful drain progress" text
+- **Health**: Wait for poll refresh, verify health dot classes update
+  - Initial state: all `.zt-health` spans have class `.zt-health-down`
+  - After ~8s (Poll interval is 5s plus async health RPC latency), reachable gateways get `.zt-health-up`
+
+Use `page.waitForFunction()` to poll DOM state rather than fixed `sleep` delays where possible. The LuCI view auto-refreshes every 5 seconds via `Poll.add()`.
+
+## Ubus RPC Patterns
+
+Login endpoint:
+```bash
+curl -X POST http://localhost:8080/ubus \
+  -H 'Content-Type: application/json' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"call","params":["00000000000000000000000000000000","session","login",{"username":"root","password":""}]}'
+```
+
+Returns `{ result: [0, { ubus_rpc_session: "...", timeout: 300, acls: {...} }] }`.
+
+Authenticated call pattern:
+```bash
+curl -X POST http://localhost:8080/ubus \
+  -H 'Content-Type: application/json' \
+  -d '{"jsonrpc":"2.0","id":2,"method":"call","params":["<TOKEN>","zt-gateway","status",{}]}'
+```
+
+## Runtime Patches
+
+Some OpenWrt packages need surgical patches for containerized environments:
+- **`/usr/share/ucode/luci/runtime.uc`**: Inject `include` into template globals in `render_ucode` so LuCI templates can call `include()`:
+  ```javascript
+  let globals = proto({ include: (name, args) => self.render_any(name, args ?? {}) }, scope ?? {});
+  ```
+- **`/usr/share/rpcd/ucode/system.uc`**: Only needed if `rpcd-mod-iwinfo` is absent. `luci-mod-admin-full` depends on `rpcd-mod-iwinfo`, so in typical LuCI installs `system.board` is natively available.
+
+
+## OpenWrt Policy Routing (ZeroTier Exit Gateway)
+
+Policy routing for subnet traffic (e.g. WIBLAN 10.11.13.0/24) requires
+three components working together:
+
+### 1. UCI Network Rule (persists across reboot)
+
+```uci
+config rule
+    option src '10.11.13.0/24'
+    option lookup '100'
+    option priority '100'
+```
+
+This is the **only** component that netifd restores automatically on boot.
+Routes in custom tables are also restored, but the `ip rule` that directs
+traffic TO those tables must be explicitly defined as a UCI `network rule`.
+
+### 2. Policy Table Routes
+
+Table 100 (main policy) and 101 (drain) must have default routes via the
+active exit gateway:
+
+```sh
+ip route replace default via <gateway_ip> dev br-zt table 100
+ip route replace default via <gateway_ip> dev br-zt table 101
+```
+
+**Critical**: Use the bridge device (`br-zt`), not the raw ZeroTier
+interface (`ztk4jpk77j`). After the ZT interface is enslaved to the
+bridge, it has no IP and `ip route replace default via <gw> dev <zt_if>`
+fails with `Nexthop has invalid gateway`.
+
+### 3. Hotplug Script
+
+Re-applies routes when the bridge comes up (e.g. after ZeroTier restart).
+Must install BOTH the table routes AND the ip rule:
+
+```sh
+# Table routes
+ip route replace default via "$active_ip" dev "$ZTG_BRIDGE" table "$ZTG_TABLE_MAIN"
+# Ip rule (commonly forgotten!)
+ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null || \
+    ip rule del from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100 2>/dev/null
+ip rule add from "$ZTG_WIBLAN_CIDR" table "$ZTG_TABLE_MAIN" priority 100
+```
+
+### Verification
+
+```bash
+# Rule exists?
+ip rule show | grep "lookup 100"
+# Table has correct default?
+ip route show table 100
+# Gateway reachable via bridge?
+ping -c 2 -I br-zt <gateway_ip>
+```
+
+## Busybox / OpenWrt Shell Gotchas
+
+1. **`ip rule replace` does not exist** in busybox `ip`. Use
+   `ip rule del ... 2>/dev/null || true; ip rule add ...` instead.
+
+2. **`awk exit` in compound if blocks**: Busybox awk mishandles `exit`
+   inside `if (...) { ...; exit }` when used with `-F` field separator
+   in a pipeline. Workaround: use `/proc/net/dev` as input instead of
+   piping from `ip -o link show`, or use multi-line awk programs.
+
+3. **Shell pipe + ssh stdin starvation**: `ssh` and `scp` consume stdin.
+   In `find | sort | while read; do ssh ...; done` loops, add `</dev/null`
+   to `ssh`/`scp` commands to prevent them from eating the pipe data.
+
+4. **UCI `network rule` vs runtime `ip rule add`**: `ip rule add` only
+   persists in the running kernel. For boot survival, write a UCI
+   `network rule` section so netifd applies it. But note: netifd may
+   not handle `ip rule replace` idempotency — always `del`+`add`.
+
+5. **Bridge ports auto-detection**: Don't rely on `ip -o link show | awk`
+   for interface detection on busybox. Prefer reading `/proc/net/dev` or
+   `/sys/class/net/` which are simpler and more portable. Always validate
+   the detected interface actually exists with `ip link show <name>`.