Most of the time, I want to do some throw away networking temporally to play
with something or to try something out. I really don't like changing all the
config on a machine just to try something. The FreeBSD documentation leans the
other way first showing you what to edit in rc.conf before maybe mentioning
that actual commands to run.
The ipfw documentation has a different problem. The example in the handbook and
online are both very verbose and very complicated. Because ipfw is normally
configured with a shell script the authors go absolutely wild with all the
features they can.
I had a hard time figuring out ipfw in-kernel NAT from these guides. Instead
here I present the simplest set of commands I could find to set up a NAT and a
little explanation to help you debug when it doesn't work.
This is based on a great email from
Allan Jude
on the
freebsd-virtualization list from 2014 that laid out the basics of this setup.
Set up Overview
For testing I want to run virtual machines and vnet jails on my laptop and give
them have access to the internet. I want a throw away NAT setup that is ready
to go quickly.
My laptop connects to my home network (and eventually the internet) over wifi.
The wifi network offers me an address in the 192.168.1.0/24 subnet. On my
laptop I want to have multiple guests. To do this we are going to use ipfw NAT
and a bridge interface. It will look something like this:
TO INTERNET
^
|
|
v
+-------+ 192.168.1.x
+-------------| wlan0 |---------------+
| +-------+ |
| ^ |
| | |
| ipfw nat |
| | |
| V |
| +---------+ |
| 10.0.4.1 | bridge0 | |
| +----+----+ |
| ^ |
| | 10.0.4.0/24 |
| ___________+_______________ |
| | | | | |
| v v v v |
| +---+--+ +--+---+ +--+---+ +--+---+|
| | jail | | vm | | jail | | ... ||
| +------+ +------+ +------+ +------+|
+-------------- laptop ---------------+
The interfaces in the jails (the b half of the epair) and the virtual machines
(the vtnet in the V) won't be visible to ipfw, but will exist in their own
world. To work around this we will use a bridge with the epairs and tap
interfaces.
Setting up ipfw NAT
We need to load the kernel modules for ipfw and the ipfw in kernel NAT. ipfw
has the frustrating default (and annoyingly different to ipf and pf) of to
dening all traffic. This default has the great property of locking you out of a
machine you are setting up remotely.
This is control by a sysctl that cannot be changed at run time, but we can
change the default behaviour with kenv before we load the module:
# kenv net.inet.ip.fw.default_to_accept=1
Now we can safely load ipfw and the in-kernel NAT.
# kldload ipfw ipfw_nat
ipfw should load enabled, if you are having trouble later on double check that
the firewall is actually enabled.
# sysctl net.inet.ip.fw.enable
net.inet.ip.fw.enable: 1
When we do NAT we are acting as a gateway between the traffic on the NATd
interface and the real interface. For any packets to be passed we need to
enable forwarding.
# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1
ipfw rule set
We need to create an IPFW NAT instance configured with the interface we want to
NAT (wlan0 in this case) and configure rules to pass all traffic from the
bridge through the NAT.
# ipfw nat 1 config if wlan0
# ipfw add 101 nat 1 ip from 10.0.4.0/24 to any out via wlan0
# ipfw add 103 nat 1 ip from any to any in via wlan0
I like to leave a gap between rules like this so I can insert an ipfw log
command for the eventual case that nothing makes sense and everything is
broken.
set up interfaces
A bridge is the center of our guest network, we will give it the default root
address that all of our guests will speak to.
# ifconfig bridge create
bridge0
# ifconfig bridge0 inet 10.0.4.1/24 up
Our jail will use an epair interface to speak to the outside world. They come
as an a and a b part, ifconfig only tells us about the a part when it clones
the interface. When we give a vnet jail an interface it is no longer visible to
the host system. An epair gives us two interfaces that act like a virtual
ethernet cable, we stick one end into the jail and the other is connected to
the bridge.
# ifconfig epair create
epair0a
Our virtual machine will use a tap interface to access the world. The tap
interface needs to be brought up. There is a helpful sysctl that is off by
default which will trigger the interface to be brought up when it is first
opened. I like to set this to one, otherwise I find myself debugging networking
inside the VM alot with little success.
# ifconfig tap create
tap0
# sysctl net.link.tap.up_on_open=1
With all the interfaces set up we need to add them to our bridge.
# ifconfig bridge0 addm epair0a addm tap0
Create jail
Never spoken about is the bsdinstall jail command. It takes a directory and
installs a jail into it. This command will ask you some questions, it would be
cool if it didn't, that would make automating jail creation in scripts much
easier for me.
# mkdir testjail
# bsdinstall jail testjail
We make our jail persist so it will stick around as we experiment. The
following command creates the jail on the host:
# jail -c name=testjail persist vnet path=testjail vnet.interface=epair0b
Now we can jexec into the jail and configure the epair. When you bring one end
of an epair up, the other end comes up, when it goes down the other end goes
down. We just need to configure an address and a default route in our jail.
# jexec testjail sh
[testjail] # ifconfig epair0b inet 10.0.4.4/24 up
[testjail] # route add default 10.0.4.1
[testjail] # ping -c 1 10.0.4.1
[testjail] # ping -c 1 192.168.1.1
[testjail] # ping -c 1 8.8.8.8
With this setup the jail can speak to our bridge, the local network and the
wider Internet.
Create and config a VM
The FreeBSD offers prebuilt virtual machine images, The latest current one is
available from a url like this:
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/FreeBSD-13.0-CURRENT-amd64.raw.xz
It would be cool if there was a latest symlink that gave you a new head VM from
one static place. The image comes xz compressed, we need to unpack it and I
like to move it to a consistent place:
# xz -d FreeBSD-13.0-CURRENT-amd64.raw.xz
# mv FreeBSD-13.0-CURRENT-amd64.raw /vms/freebsd-current
bhyve requires we load the vmm kernel module, with that we can use the
excellent vmrun.sh script to launch our vm.
# kldload vmm
# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024 -t tap0 -d /vms/freebsd-current freebsd-current
Once that comes up you can log in and do some manual config.
[vm] # ifconfig vtnet0 inet 10.0.4.5/24 up
[vm] # route add default 10.0.4.1
[vm] # ping 8.8.8.8
For DNS in both the jail and the virtual machines I have to manually set up the
name server local from my network.
/etc/resolv.conf
search lan
nameserver 192.168.1.1
This won't be valid as I move to other networks, but I am sure I will remember
after only a little confusion and debugging.
Conclusion
That is all it takes. The NAT configuration is 3 firewall rules and enabling
forwarding. None of this is persistent and that isn't great practice for a
production environment, but it you just want to experiment with ipfw and NAT,
or spin up a VM for today knowing how to do this in a non-persistent way is
really helpful.
