# TailMux - OrbStack Tailnet Lab

Date: 2026-07-01

Status: Test harness for the service-tunnels work
([09-service-tunnels](09-service-tunnels.md)) and for TailMux's routing /
isolation semantics.

## Purpose

A disposable lab, built from OrbStack Linux VMs, that reproduces every routing
case TailMux must handle — including the **Tailscale app connector** case that
`tsnet` cannot reach directly (the reason Tier B / `ssh -L` exists). Do this on
a **dedicated test tailnet**, never on work, and never on your personal tailnet.

### Why a dedicated tailnet (not personal)

App-connector and split-DNS configuration is **tailnet-global**, not per-device.
Adding it to your personal tailnet would change DNS resolution for every device
on it. A throwaway tailnet lets you break ACLs/DNS/routes freely, doubles as a
clean second profile for isolation tests, and can be shared with a colleague.
Creating a second tailnet needs a separate identity (another Google/GitHub
account or a GitHub-org tailnet) — one-time friction.

### The golden rule

Always test **through TailMux** (`tailmux connect` / `tailmux tunnel` / the
proxy), never from the Mac shell. All OrbStack VMs share the host LAN, so from
the Mac you could reach a VM's IP directly and get a false pass. TailMux's
`tsnet` nodes use their own netstack and only reach the tailnet + accepted
routes — which is exactly the isolation we are validating.

## Topology

```text
                 test tailnet (dedicated)
   ┌───────────────┬───────────────┬──────────────────┐
   │  connector    │  subnet-router │     bastion       │
   │  (tsnet node) │  (tsnet node)  │  (tsnet node,     │
   │  app connector│  advertises    │   sshd)           │
   │  *.internal.  │  db subnet     │                   │
   │   example     │                │                   │
   └──────┬────────┴───────┬────────┴────────┬──────────┘
          │  OrbStack LAN   │                 │
          └────────┬────────┴─────────────────┘
                   ▼
                ┌──────┐   db  (NO Tailscale)
                │  db  │   postgres:5432  +  http:8080
                └──────┘   the "VPC-only" resource

   TailMux profiles:  test  -> test tailnet (the 4 VMs)
                      personal -> your existing tailnet (isolation tests only,
                                  no DNS changes)
```

| VM | Tailscale? | Role | Exposes |
| --- | --- | --- | --- |
| `db` | no | VPC-only resource | `postgres:5432`, `http:8080` |
| `connector` | yes (`tag:lab-connector`) | app connector for `*.internal.example` → `db` | reproduces the app-connector DB |
| `subnet-router` | yes | advertises `db`'s subnet | makes `db` reachable via route |
| `bastion` | yes | tailnet node + `sshd`, reaches `db` on the LAN | Tier B `ssh -L` target |

`connector` and `subnet-router` are kept **separate on purpose**: on the *same*
`db` you can compare "reachable via route (Tier A works)" against "reachable
only via connector domain (Tier A fails, Tier B needed)".

## Prerequisites

- OrbStack installed on the Mac.
- A dedicated test tailnet + admin access to its policy file (ACLs) and DNS.
- TailMux built/installed with a `test` profile pointing at the test tailnet.

## Build

Placeholders: `<tailnet>` = your test tailnet's `*.ts.net` name;
`internal.example` = the fake internal domain; `10.99.0.0/24` = `db`'s subnet
(use the real OrbStack subnet you get — see step 1).

### 0. Provisioned instance (2026-07-03)

The four VMs are created and OS-provisioned. Concrete values for this run:

| Placeholder | This lab |
| --- | --- |
| `db`'s subnet | `192.168.139.0/24` (OrbStack) |
| `db` IP | `192.168.139.212` |
| `connector` IP | `192.168.139.111` |
| `subnet-router` IP | `192.168.139.149` |
| `bastion` IP | `192.168.139.82` |
| SSH user (bastion) | `claudioquaglia` |
| connector domain | `db.internal.example` → `db` |

Already done by provisioning (the OS side of steps 2–5):

- `db`: postgres on `:5432` (trusts `192.168.139.0/24`, database `labdb`), http
  on `:8080`. No Tailscale.
- `bastion`: `tailscale` + `sshd` active, your `~/.ssh/*.pub` installed in
  `authorized_keys`. Reaches `db:5432` on the LAN.
- `subnet-router`: `tailscale` + IP forwarding enabled. Reaches `db:5432`.
- `connector`: `tailscale` + IP forwarding + `dnsmasq` resolving
  `db.internal.example` → `192.168.139.212` (bound to eth0 + tailscale0
  via `bind-dynamic`, deliberately not loopback).

Remaining — **interactive**, do in this order (needs your Tailscale auth + admin
console):

1. Apply the policy file from step 5 (tagOwners / autoApprovers / grants /
   nodeAttrs). `tag:lab-connector` must exist before the connector can advertise
   it.
2. Bring the nodes up (each prints an auth URL to open):
   ```bash
   orb -m bastion       sudo tailscale up
   orb -m subnet-router sudo tailscale up --advertise-routes=192.168.139.0/24
   orb -m connector     sudo tailscale up --advertise-connector --advertise-tags=tag:lab-connector
   ```
3. Approve the subnet-router route (Machines → subnet-router → Subnet routes),
   unless an `autoApprovers` rule covers it.
4. DNS → Nameservers: add a split-DNS nameserver for `internal.example` pointing
   at the connector's **tailnet** IP (`orb -m connector tailscale ip -4`).
5. TailMux `test` profile: ensure `suffixes` include `.internal.example` and the
   test tailnet's `.ts.net`.

### 1. Create the VMs

```bash
orb create ubuntu db
orb create ubuntu connector
orb create ubuntu subnet-router
orb create ubuntu bastion

# note each VM's OrbStack IP; you'll need db's IP and its /24
orb list
ip -4 addr show            # run inside a VM (orb -m db) to see its address
```

Enter a VM with `orb -m <name>` (or `ssh <name>@orb`). Commands below are run
inside the named VM.

### 2. `db` — the VPC-only resource (no Tailscale)

```bash
# orb -m db
sudo apt-get update && sudo apt-get install -y postgresql netcat-openbsd python3
sudo sed -i "s/#listen_addresses.*/listen_addresses='*'/" /etc/postgresql/*/main/postgresql.conf
echo "host all all 10.99.0.0/24 trust" | sudo tee -a /etc/postgresql/*/main/pg_hba.conf
sudo systemctl restart postgresql
# simple HTTP service to also exercise proxy/HTTP paths
nohup python3 -m http.server 8080 >/tmp/http.log 2>&1 &
```

Do **not** install Tailscale here. This is what makes `db` reachable only via
the connector / subnet-router / bastion.

### 3. `bastion` — tailnet node + sshd (Tier B target)

```bash
# orb -m bastion
curl -fsSL https://tailscale.com/install.sh | sh
sudo apt-get install -y openssh-server
sudo systemctl enable --now ssh
sudo tailscale up   # authenticate into the TEST tailnet
```

Confirm the bastion reaches `db` on the LAN: `nc -vz 10.99.0.<db> 5432`.
Add your SSH public key to the bastion user's `~/.ssh/authorized_keys` so Tier B
can use your agent/keys.

### 4. `subnet-router` — advertise `db`'s subnet (Tier A route path)

```bash
# orb -m subnet-router
curl -fsSL https://tailscale.com/install.sh | sh
echo 'net.ipv4.ip_forward = 1'        | sudo tee /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf
sudo tailscale up --advertise-routes=10.99.0.0/24
```

Approve the route in the admin console (Machines → subnet-router → Edit →
Subnet routes), or via `autoApprovers`. The TailMux `test` profile must set
`accept_routes: true` (maps to `RouteAll`).

### 5. `connector` — app connector for `*.internal.example` (Tier A failure)

Node side:

```bash
# orb -m connector
curl -fsSL https://tailscale.com/install.sh | sh
echo 'net.ipv4.ip_forward = 1'        | sudo tee /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

# resolver that maps the fake internal domain to db (split-DNS backs this)
sudo apt-get install -y dnsmasq
echo "address=/db.internal.example/10.99.0.<db>" | sudo tee /etc/dnsmasq.d/lab.conf
sudo systemctl restart dnsmasq

sudo tailscale up --advertise-connector --advertise-tags=tag:lab-connector
```

Tailnet policy file (admin console → Access controls):

```json
{
  "tagOwners": {
    "tag:lab-connector": ["autogroup:admin"]
  },
  "autoApprovers": {
    "routes": {
      "0.0.0.0/0": ["tag:lab-connector"],
      "::/0":      ["tag:lab-connector"]
    }
  },
  "grants": [
    { "src": ["*"], "dst": ["*"], "ip": ["*"] },
    { "src": ["autogroup:member"], "dst": ["tag:lab-connector"], "ip": ["tcp:53", "udp:53"] }
  ],
  "nodeAttrs": [
    {
      "target": ["*"],
      "app": {
        "tailscale.com/app-connectors": [
          {
            "name": "lab-internal",
            "connectors": ["tag:lab-connector"],
            "domains": ["*.internal.example"]
          }
        ]
      }
    }
  ]
}
```

DNS (admin console → DNS → Nameservers): add a **split-DNS** custom nameserver
for `internal.example` pointing at the `connector` node's tailnet IP (its
dnsmasq). This is what makes the domain resolvable and observable by the
connector — and it is the exact piece that `tsnet` does not consume, which is
why Tier A fails on it.

### 6. TailMux `test` profile

In your TailMux config (see [00-getting-started](00-getting-started.md)):

```yaml
profiles:
  test:
    backend: tsnet
    accept_routes: true
    suffixes:
      - .<tailnet>.ts.net
      - .internal.example
```

## Test matrix

Run everything through TailMux. Expected results in the last column.

| # | Case | Command | Expected |
| --- | --- | --- | --- |
| 1 | Tier A — tailnet node | `tailmux connect bastion.<tailnet>.ts.net 22` | connects (SSH banner) |
| 2 | Tier A — subnet route | `tailmux tunnel --listen 127.0.0.1:15432 10.99.0.<db>:5432` then `psql -h 127.0.0.1 -p 15432` | connects to `db` |
| 3 | Tier A — app connector (warm service) | `curl -x socks5h://127.0.0.1:43101 http://db.internal.example:8080/` with subnet-router down | **succeeds** via the connector's `/32` (verified 2026-07-03: HTTP 200, 3/3) |
| 3b | Cold-start CLI vs app connector | `tailmux connect db.internal.example 5432` on a cold profile | fails (`did not reach Running` / DNS deadline) — the cold-start hazard, NOT a connector limit |
| 4 | Tier B — `ssh -L` via bastion | `ssh -o ProxyCommand="tailmux connect %h %p" -N -L 127.0.0.1:15432:db.internal.example:5432 <user>@bastion.<tailnet>.ts.net` then `psql -h 127.0.0.1 -p 15432` | connects to `db` through the bastion |
| 5 | Deny — unclassified `.ts.net` | `tailmux connect something.unknown.ts.net 443` | denied, no route |
| 6 | Deny — raw IP (SOCKS5) | dial the SOCKS5 proxy to a raw IP | denied (`ErrIPLiteral`) |
| 7 | Isolation — no cross-profile | route a `test` host while only `personal` is up | denied, no fallback |
| 8 | No fallback after failure | `sudo tailscale down` on the target's node, retry | denied, does not use another profile |
| 9 | Cold start | first `tailmux connect` on a cold profile | may report `did not reach Running`; warm service does not |

Key results (2026-07-03 run): case 3 **passes** — `tsnet` traverses the app
connector when split-DNS is set, ACLs allow, and the service is warm. The
failures that motivated this lab were reproduced as case 3b (cold-start) and
the ACL gotcha below, not as a connector limitation. Tier B remains the
fallback for true jump-host-only targets and was proven end-to-end (case 4,
psql via bastion through the warm service).

## Gotchas

- **The policy file REPLACES the tailnet's default allow-all ACL.** If you paste
  a policy that only contains the connector's `tcp:53/udp:53` grant, ALL other
  traffic between nodes is denied — every TCP dial fails while `tailscale ping`
  still works (disco pings bypass ACLs, so ping is NOT proof the ACL is fine).
  Keep an explicit allow-all grant (`{"src":["*"],"dst":["*"],"ip":["*"]}`) in
  the lab policy, as in the snippet above. This exact mistake burned a test
  session on 2026-07-03.

- **Split-DNS + app connector is the fiddly part.** If case 3 *succeeds*, the
  connector likely isn't the active path — check the split-DNS nameserver is
  restricted to `internal.example` and that no subnet route also covers `db`
  (keep `connector` and `subnet-router` distinct so the paths don't overlap).
- **IP forwarding** must be enabled on `connector` and `subnet-router`, and
  their routes approved.
- **Tailscale needs `/dev/net/tun`** for kernel networking (subnet router / app
  connector). OrbStack VMs provide it; if a node comes up in userspace-only
  mode, subnet/connector routing won't work.
- **`db` must not run Tailscale.** If it joins the tailnet it becomes a normal
  node and case 3 stops reproducing the failure.

## Teardown

```bash
orb delete db connector subnet-router bastion
```

Then remove the lab nodes from the admin console and revert the policy-file /
DNS entries (or just delete the throwaway tailnet).
