I recently built a storage server (SAN/NAS/whatever) with Nas4Free on a Supermicro 6037R-E1R16N. If you’ve ever built a SAN, you’ve seen one of these. Lots of disks in a ZFS array, with a filesystem exported via NFS or a volume exported via iSCSI. After almost a month of operation, I discovered that Nas4Free defaults to having no swap space. And this means your storage server will not be stable.
tl;dr? Here’s how to fix it:
- Login to the Nas4Free web interface.
- Navigate to “Disks” -> “ZFS” -> “Volumes”.
- Create a ZFS Volume called “swap”. For size, “2x RAM” is the standard. Otherwise use defaults.
- Navigate to “System” -> “Advanced” -> “Swap”.
- Set “Type” to “Device”, and “Device” to “/dev/zvol/tank/swap”, where “tank” is your ZFS pool name and “swap” is the name of the volume you just created as a swap device.
If you don’t already have ZFS configured, first create your ZFS pool as normal. Then follow the above steps.
Nas4Free allows you to install “embedded” or “full” software installations. I chose “embedded”, because this makes upgrades very easy, and it is simple to install this way on a USB flash device. This means you can dedicate all SATA ports to ZFS storage and not the operating system. With the embedded installation, Nas4Free does not create a swap space. Which means it’s up to the user to do this. It’s possible this isn’t an issue on the “full” installation. I don’t know; I haven’t tried it.
Why is swap necessary? After all, if you have a large amount of RAM, you shouldn’t need swap. And if you use ZFS, you should always have a large amount of RAM. That’s the ideal situation, but in practice it doesn’t work. Why? Because ZFS uses any available memory for cache.
Under FreeBSD, this is called wired memory. When ZFS uses wired memory as cache, it should be reclaimed for other uses when it’s needed. That’s certainly what happens under Linux, where disk (read) cache is reported as “Cache memory” in the above graph.
I have found this isn’t the case in practice. Instead, when memory gets tight, FreeBSD kills processes it really shouldn’t:
Apr 30 12:49:45 san1 kernel: pid 69430 (istgt), uid 0, was killed: out of swap space
That’s right. Run ZFS for long enough with no swap, and eventually your iSCSI daemon will be killed. Which means your SAN isn’t accessible any longer. It could also kill the NFS server if you are using the userspace NFS daemon, which is what Nas4Free does.
from which Nas4Free is forked a similarly-named commercial product, doesn’t have this problem. It defaults to using a certain amount of each disk for swap, which means the average user simply won’t see this problem. This may be a moot point, since the average user in this situation should be a highly-trained engineer. But it happened to me, and I’m documenting it here so it doesn’t happen to you.
Thanks to the fine people of the Nas4Free forums for providing additional information.