RAM and Recklessness: The Network Nobody Asked For

Part 2 of Building a Diskless Datacenter

When you have three servers with DPUs, a 100G switch, and a routing addiction, you don't build a flat network. You build something with VLANs, BGP, and a naming scheme that makes you feel like you're running a real datacenter. Then you spend three weeks debugging why packets aren't going where you think they should.

The Supernet

The first real design decision was the IP addressing. I carved out 10.42.64.0/18 — a clean supernet that gives me 64 /24 subnets to play with, all routable with a single rule on the upstream router.

The trick: VLAN IDs match the third octet. VLAN 67 is always 10.42.67.0/24. VLAN 68 is 10.42.68.0/24. No lookup tables, no spreadsheets. If you know the VLAN, you know the subnet.

Eight active VLANs, eight reserved for future use. The whole block routes upstream through the CCR2116 with one static route.

The RDS as Core Switch

The MikroTik Rose Data Server sits at the center of everything. Its port allocation:

• 4x 25G SFP28 → one per DPU (c4140 on .92, r640 on .82, r740xd on .42)
• 4x 10G SFP+ → host connections and uplinks
• 1x 100G QSFP28 → Arista uplink for legacy network

The DPU ports carry tagged traffic for VLANs 67 (control) and 68 (storage), plus VLAN 65 untagged for PXE boot. Each DPU's OVS bridge trunks these VLANs to the host's virtual function, so the NixOS host sees tagged VLAN interfaces even though the physical port is on the DPU.

This is where it gets weird: the RDS is also the PXE server and the NVMe-oF target. Your boot images? Served from the RDS over HTTP on VLAN 65. Your persistent volumes? NVMe/TCP targets on the RDS, accessible via VLAN 68. The switch is the storage.

DPU-Centric Routing

This was ADR-003 and it's the decision that made everything else possible — and caused the most debugging.

Traditional setup: hosts have their own network ports and route directly. Our setup: all host traffic flows through the DPU. The host's only network path to the outside world is through the BlueField-2's OVS bridge.

Host (NixOS) → VF (virtual function) → DPU OVS → Physical 25G port → RDS

The host's default route points to the DPU's VLAN 67 gateway. This means:

• If the DPU goes down, the host loses all network connectivity
• All traffic is inspectable and filterable at the DPU layer
• The host doesn't need any physical network configuration

We chose this deliberately. The DPU is the network boundary for each host. In a production SmartNIC deployment, this is where you'd run firewall rules, encryption, and traffic shaping — all offloaded from the host CPU.

In practice, it meant that debugging network issues required SSH-ing to the DPU first, because the host couldn't tell you anything useful about what was happening on the wire.

The OVS Configuration Saga

Each DPU runs Open vSwitch with a bridge configuration that looks roughly like:

br-hfp0 (bridge)
├── p0 (physical 25G port) — trunk VLANs 65,67,68; native VLAN 65
├── pf0hpf (host representor) — trunk VLANs 67,68
└── management port — VLAN 67

Getting this right took iterations. The p0 port needed native-untagged mode for VLAN 65 (PXE boot uses untagged frames), but the host representor ports needed tagged-only access to the control and storage VLANs.

The fun discovery: OVS port names on the BlueField-2 aren't what you'd expect. The host-facing representor ports are named pf0hpf (physical function 0, host port function), not something reasonable like eth0 or host0. Miss a letter and your VLAN trunk goes nowhere.

The VLAN 64 Migration

Management (VLAN 64) was an early headache. The iDRACs and DPU management ports were originally on a legacy subnet (10.42.241.0/24) connected through the old Brocade switch stack. Migrating them to the new VLAN 64 on the RDS meant:

1. Updating iDRAC IPs via the existing management interface
2. Moving DPU management ports to the new subnet
3. Updating all documentation and tools
4. Praying nothing lost connectivity mid-migration

The key decision: management traffic stays on the CCR's 1Gbit ports and never touches the RDS. Out-of-band management is completely separate from the high-speed data plane. If the RDS explodes, I can still reach every iDRAC.

Storage Network Isolation

VLAN 68 is dedicated to storage — NVMe/TCP traffic between hosts and the RDS. This is critical because NVMe/TCP is latency-sensitive and bandwidth-hungry. Mixing it with pod traffic on VLAN 67 would be asking for trouble.

The storage path:

Host initrd → VLAN 68 interface → RDS NVMe/TCP target (10.42.68.1:4420)

Later, when we built the CSI driver, it would provision volumes on the RDS via SSH (on the management network) and present them to pods via NVMe/TCP (on the storage network). Two paths, two purposes, complete isolation.

Lessons Learned

1. VLAN = subnet naming is a gift — When you're debugging at 2 AM, not having to look up what VLAN 67 means saves real time.

2. DPU-centric routing is powerful but unforgiving — One OVS misconfiguration and the host is an island. Always have iDRAC access as a backup path.

3. Separate management from data — When the data plane is broken, you need a way to reach the machines. Don't route management through the same switch you're debugging.

4. The RDS is underrated — A device that's a 100G switch, NVMe-oF target, and PXE server simultaneously shouldn't work this well. MikroTik's RouterOS is bizarre but capable.