Towonel Tunnel¶
External traffic reaches the cluster through towonel, a self-hosted tunnel that replaced Cloudflare Tunnel (cloudflared) in July 2026. An agent in the cluster dials out to a hub on a VPS, so the router still needs no inbound port forwards and the home IP stays unpublished.
The tunnel has two halves:
| Half | Runs on | Managed by |
|---|---|---|
| Hub + edge | ovh-vps (tunnel.wibrow.dev) | Ansible role towonel-hub — see the runbook in ansible/README.md |
| Agent | pitower cluster | kubernetes/apps/pitower/networking/towonel-agent/ |
Architecture¶
flowchart LR
User((User))
subgraph VPS["ovh-vps — tunnel.wibrow.dev"]
Caddy[Caddy L4<br/>SNI demux :443]
Edge[towonel edge]
Hub[towonel hub<br/>control API]
end
subgraph Cluster["pitower"]
Agent[towonel-agent<br/>2-4 replicas]
EE[envoy-external]
Apps[Applications]
end
User -->|"HTTPS *.wibrow.dev<br/>(DNS: unproxied CNAME)"| Caddy
Caddy -->|"tenant SNI"| Edge
Caddy -->|"SNI = tunnel.wibrow.dev"| Hub
Edge <-.->|"outbound tunnel"| Agent
Agent -->|"HTTPS :443"| EE
EE --> Apps
classDef vps fill:#f59e0b,stroke:#d97706,color:#000
classDef gw fill:#7c3aed,stroke:#5b21b6,color:#fff
class Caddy,Edge,Hub vps
class EE gw Traffic flow¶
- User requests
https://myapp.wibrow.dev. - Cloudflare DNS returns the unproxied (grey-cloud) CNAME to
tunnel.wibrow.dev, which resolves to the VPS. Cloudflare is authoritative DNS only — it is not in the data path. - Caddy on the VPS peeks the TLS ClientHello SNI on
:443without decrypting it, and routes tenant hostnames to the towonel edge. - The edge forwards the connection over the already-established outbound tunnel to
towonel-agentin the cluster. - towonel-agent proxies to
envoy-externalon:443. - envoy-external matches the HTTPRoute hostname and routes to the application.
TLS is terminated by envoy-external inside the cluster — the VPS does SNI passthrough and never sees plaintext.
Why Caddy is in front of towonel
The hub's own ACME issuance uses TLS-ALPN-01, which only ever validates on port :443 — a port that otherwise belongs exclusively to the edge, which has no ACME awareness. Caddy's L4 SNI demux lets the validator connection land on a listener that understands ACME. The full write-up is in the towonel-hub runbook in ansible/README.md.
Agent configuration¶
The agent is stateless. It is given an invite token and a list of hostname → origin mappings:
env:
TOWONEL_AGENT_HEALTH_LISTEN_ADDR: 0.0.0.0:9090
TOWONEL_AGENT_SERVICES: |
[
{"hostname":"*.wibrow.dev","origin":"envoy-external.networking.svc.cluster.local:443"}
]
TOWONEL_INVITE_TOKEN:
valueFrom:
secretKeyRef:
name: towonel-agent-secret
key: TOWONEL_INVITE_TOKEN
A single wildcard service covers every public hostname, so adding a public app needs no tunnel change — only an HTTPRoute with parentRefs to envoy-external.
The deployment runs 2 replicas with an HPA to 4 on 75% CPU (hpa.yaml), non-root with a read-only root filesystem and all capabilities dropped.
The invite token embeds the hub URL
TOWONEL_INVITE_TOKEN (Infisical, /networking/towonel-agent/) hard-codes TOWONEL_HUB_PUBLIC_URL. Changing the hub URL invalidates the token — it must be reissued from the hub and the agent restarted. Each invite is a new tenant identity; the old tenant is orphaned and should be cleaned up with towonel tenant remove.
DNS¶
Subdomains are unproxied CNAMEs to the hub — Cloudflare proxying would break the SNI passthrough the edge depends on:
- dnsName: "external.wibrow.dev"
recordType: CNAME
targets: ["tunnel.wibrow.dev"]
providerSpecific:
- name: external-dns.alpha.kubernetes.io/cloudflare-proxied
value: "false"
- dnsName: "*.wibrow.dev"
recordType: CNAME
targets: ["tunnel.wibrow.dev"]
providerSpecific:
- name: external-dns.alpha.kubernetes.io/cloudflare-proxied
value: "false"
The wildcard means an unmatched subdomain still reaches the edge and hits the envoy-external fallback 404 route rather than returning NXDOMAIN.
The apex is not on the tunnel
wibrow.dev is served entirely by a Cloudflare Worker route at the edge and has no origin. It keeps a proxied placeholder record (AAAA 100::, the IPv6 discard prefix) purely so the Worker route has a proxied hostname to attach to — see kubernetes/apps/pitower/networking/external-dns/dnsendpoint.yaml.
Health and troubleshooting¶
The agent serves /healthz on :9090, used for all three probes.
# Agent status
kubectl get pods -n networking -l app.kubernetes.io/name=towonel-agent
kubectl logs -n networking -l app.kubernetes.io/name=towonel-agent --tail=50
# Hub status (on the VPS)
systemctl status towonel-hub
docker logs -f towonel
# Is the hub reachable and presenting a valid cert? (run off-VPS)
curl -v https://tunnel.wibrow.dev/v1/health
# Confirm DNS is unproxied — the answer must be the VPS IP, not a Cloudflare IP
dig +short external.wibrow.dev
Everything public returns 5xx
Check the agent's tunnel is established (kubectl logs), then that the hub is up on the VPS. Because the agent dials out, a hub restart drops every route until the agent reconnects.
A hostname returns 404 from Envoy
The tunnel is fine — the wildcard delivered the request and envoy-external had no matching HTTPRoute. Check the app's HTTPRoute hostnames and parentRefs.
Cloudflare error page instead of the app
The DNS record got proxied. external-dns applies --cloudflare-proxied globally, so every towonel record needs the explicit cloudflare-proxied: "false" providerSpecific override.