A few months ago I got gripped with server fever. I'd had a raspberry pi for a while, but I could never find a meaningful use for it. I'd tried running plex, etc, but the pi3 would choke after a bit and sputter out.
One day I decided enough was enough: it was time to build a more beefy server. (and I found a compelling use for the pi!)
Current FreeNAS server information
I had a few goals I wanted to achieve:
- be relatively future-proof
- host a plex server
- be able to spin up multiple VMs for various reasons like testing ansible scripts
- potentially host games for me and my brother
If all you want to do is host Plex/jellyfish, a raspberry pi should suffice. As long as your media isn't hvec x265. Before I built this server I had two 2tb drives plugged into my raspberry pi, and other content was fine for the most part.
I don't have space in my home for a rack for a server, due to both the size and noise. That immediately eliminated a lot of build paths that I saw.
The ideal spot for the server was my living room, so I needed a relatively low profile server- and a quiet one.
The remaining requirements were:
- at least 8 hard drive bays
- a relatively modern/beefy cpu
- low(er) power consumption
I felt like 8 drives (+1 or 2 ssds) was enough for future expansion, even with RAID-10 taken into account. I went with a mirrored structure so I could replace drives without heavy rebuild times, and so I could easily expand the pool if necessary. I started with 6 drives, so I have a zpool with 3 vdevs of 2 drives each.
What I thought was going to be a relatively simple task turned into a huge amount of research... I spent most of my time on iXsystems' community forum and /r/homelab on reddit. The people on iXsystems' forum were very helpful and detailed. However, most of the information for iXsystems is based around used intel boards. Much less material is available if you're like me and are interested in newer AMD cpus.
So how did I narrow it down? I did what I always do: make a spreadsheet. There are simply way too many options presented to figure what cpu/mobo to choose without spreadsheet analysis. Without further ado:
- case: Fractal Node 804 Micro Atx Case
- boot ssd (nvme): WD Black NVMe M.2 2280 250Gb PCI-Express 3.0 x4 SSD WDS250G2X0C
- ssd: SAMSUNG 860 EVO Series 2.5" 500GB MZ-76E500B/AM
- intake fans x4: Noctua NF-F12 WPM 120mm
- exhaust fans x2: Noctua NF-A14 PWM 140mm
- cpu cooler: Noctua NH-L12S 70mm Low-Profile CPU Cooler
- ups/apc: https://www.newegg.com/apc-bx1500m-5-x-nema-5-15r-5-x-nema-5-15r/p/N82E16842301561?Description=APC%20BX1500M%20Back-UPS%20Pro%201500VA%2f900W&cm_re=APC_BX1500M_Back-UPS_Pro_1500VA%2f900W--42-301-561--Product
- psu: Seasonic FOCUS Plus Series SSR-650FX 650W
The total comes out to around $632. I don't know why anyone would build a server without getting some sort of UPS/APC as well.
- cpu: AMD Ryzen 5 3600 - 3.6Ghz/4.2Ghz 6-core
- mobo: ASRock X570M Pro4 AM4 MicroAtx
- ram x3: Crucial 16GB DDR4-2666 ECC UDIMM CT16G4WFD8266
The total was $676 for these parts, with $291 for the memory alone. DDR4 ram with ecc isn't cheap. For comparison, a Xeon E5-2650 V2 goes for around $70 on ebay, with significantly cheaper DDR3 ram.
NoteI've used the ryzen setup for 3+ months now with no issues.
So I was already at $1,300, before purchasing hard drives. I bought 6 Western Digital Red 4TB drives initially, @ $100. Later on I got 2 more drives, and another stick of ram.
Altogether, it was $1300 for the parts + $800 for the drives, or ~$2100 in total.
I didn't go with a used intel board because of the higher power usage and less powerful specs. This made my build much more expensive, but also more future proof.
Why jails over docker? Convenience, I guess. It's far easier for me to muck around in iocage jails than it is to inspect and debug docker containers. Unfortunately this does mean if I lose my server, I'd have to recreate all those jails... but setting them up in the first place was pretty easy once I learned how rc.d worked.
Services that I run:
backblaze -> data backup bazarr -> grabs subtitles in multiple languages, hooks into radarr/sonarr caddy -> reverse proxy to some of these services calendar -> radicale kitana -> ui for plex plugins mailbackup -> offlineimap + cron node_exporter -> feeds into prometheus on the raspberry pi paperless -> digital document store plex -> media catalogue + streaming radarr -> monitors local shows for for bazarr subtitle grabbing sonarr -> monitors local movies for for bazarr subtitle grabbing syncthing -> seamlessly sync files between computers thelounge -> modern irc client. I used to use znc+weechat, but I got tired of weechat's ux. thelounge is simple and pretty.
Several services use databases, so I elected to set aside jails for them.
mc_main mc_creative mc_survival
I also run 3 instances of minecraft via PaperMC, a high performance fork of Spigot. The main jail runs Waterfall (a fork of bungeecord) along with a hub instance. Waterfall acts as a proxy and lets one access multiple servers within a network.
My docker services:
andrewzah/gollum -> a wiki with changes automatically git pushed -> a fork of gollum with user logins insekticid/docker-piwiki (matomo) -> self-hosted analytics (tracking andrewzah.com) -> respects requests to not track user radhifadlillah/shiori -> self-hosted website backup, similar to archive.web huginn/huginn -> self-hosted, more powerful version of IFTTT cwspear/docker-local-persist-volume-plugin -> allows local volume mounts in portainer/rancheros portainer/portainer
Current FreeNAS memory graph
A Use for the Raspberry Pi
Having metrics and a dashboard is awesome, but if my server crashes, I no longer have access to the metrics... So they have to be run somewhere else! This is where the raspberry pi comes in- it just sits on my local network ingesting traffic.
Unfortunately, node_exporter doesn't seem to export hdd temperature values, which is pretty important. I'll probably have to write a simple script to pull those values and host another metrics target for prometheus.
Grafana with node_exporter metrics from FreeNAS
FreeNAS has extensive documentation. and *BSDs are nice. ZFS and RAID are nice. Free Software is nice.
If I didn't use docker so heavily I would likely use FreeBSD or OpenBSD for my personal computers as well instead of Debian.
Overall the process of building a server was pretty fun, but I'm glad I'm done with that for the foreseeable future. I didn't self-host that much when I began, but once I started adding services I started thinking about everything I could self host.