I migrated all my self-hosted services to FreeBSD

After having previously sworn my loyalty to OpenBSD, I have since come across a very persuasive article regarding the use of FreeBSD for hosting services. After a bit of thought, I decided it was a good time to improve my self-hosting situation.

How it started

My situation going into this migration was not exactly good. I had a Proxmox server which was set up quite well and hosting most of my services, but some of the earliest things I had set up were running on a bare metal Debian installation with no containerization and an ordinary ext4 file system on a single SSD. It was a ticking time bomb and needed to be replaced.

The plan

Phase one: make a FreeBSD VM on the Proxmox machine to learn how to set up jails with bastille.

Phase two: move all services hosted on the Debian box to jails in this VM, except for Jellyfin, which has system-specific configuration for hardware transcoding acceleration.

Phase three: install FreeBSD on the old Debian box, create a bhyve VM for Jellyfin, then migrate all the jails from the FreeBSD test VM to the new server.

Phase four: migrate all the VMs from Proxmox to jails on the new server where applicable, or convert them to bhyve VMs.

Phase five: install FreeBSD on the Proxmox box, and move jails/VMs back over to it as appropriate.

How it went

Thanks to the flexibility of zfs, this plan went almost entirely painlessly. The only hitch I ran into was trying migrate an OpenBSD VM to bhyve, which failed because bhyve does not support booting OpenBSD from an MBR. I could have figured out how to use grub to boot it, but it was easier to just rebuild the VM using uefi.

Why this is so so so so so so nice to have

There are two main reasons why this is a big improvement: zfs and jails.

zfs is a truly powerful file system that to my knowledge is unmatched by its competitors. Yeah, btrfs exists, but I have heard far too many horror stories of losing entire volumes because of btrfs bugs that I don’t think I will ever trust it. zfs just works, and really the only problem with it is the CDDL license which makes it a pain to set up on Linux, especially if you want your root file system to be on zfs. Being able to snapshot and replicate the root file system means that disaster recovery is trivial: just install FreeBSD again and then zfs send the backup over, and you’re done.

Migrations are also trivialized. I can just zfs send entire VMs or jails between hosts and be migrated in a matter of seconds, limited only by network speeds. In fact, the major contributor to the ease of migrating VMs off of Proxmox is the fact that I was using zfs on Proxmox for the VMs’ disk images.

Jails are also a major improvement. They offer the flexibility and process isolation of virtualization without the overhead of running a totally different operating system with hardware emulation layers. I know containers exist on Linux, but jails are first class on FreeBSD, not some third-party thing you bolt on. Not to mention, I don’t have to fuss with choosing a distro. It’s all FreeBSD, and since I use thin jails, I can apply patches to all of my jails at once and save disk space by not having duplicate files.

I also really appreciate that the managers I installed for jails and VMs are implemented as a set of shell scripts. Everything I’m doing is supported by the base FreeBSD system; these management scripts are just refined forms of what I would have written for myself anyway.

Give it a try

Seriously, the next time you are thinking of standing up a server, try out FreeBSD. Install bastille and vm-bhyve and give it a go. The learning curve is quite small, and you’ll thank yourself later.

I think the only place I could go from here is back to OpenBSD if they ever implement these sorts of features, but I don’t see that happening any time soon.

Where I’m not using FreeBSD

OpenBSD is still king of security — they are downright paranoid. I still use OpenBSD for my home router, and I use it for VMs where I need to make sure things are locked down as much as they could be.

I still see myself using GNU/Linux on the desktop for the foreseeable future. The BSDs have lots of the desktop software, but the desktop experience really seems to be an afterthought, and that’s fine. I’d rather they focus on making really solid Unix systems. The jank of Linux is more tolerable on desktop systems anyway.

And of course Linux has better hardware support generally than the BSDs due to its popularity, so there are still some edge cases where Linux is the only good option, but thankfully those are few and far between.

Finally, I still like NetBSD as a fallback option for very old or limited hardware, or in cases where I need support for legacy protocols that other OSes have declared to be obsolete. It’s a tidy little system (though not as tidy as OpenBSD in my opinion), and their dedication to legacy support is truly inspiring.