Enable swap on Nas4Free

I recently built a storage server (SAN/NAS/whatever) with Nas4Free on a Supermicro 6037R-E1R16N. If you’ve ever built a SAN, you’ve seen one of these. Lots of disks in a ZFS array, with a filesystem exported via NFS or a volume exported via iSCSI. After almost a month of operation, I discovered that Nas4Free defaults to having no swap space. And this means your storage server will not be stable.

tl;dr? Here’s how to fix it:

  1. Login to the Nas4Free web interface.
  2. Navigate to “Disks” -> “ZFS” -> “Volumes”.
  3. Create a ZFS Volume called “swap”. For size, “2x RAM” is the standard. Otherwise use defaults.
  4. Navigate to “System” -> “Advanced” -> “Swap”.
  5. Set “Type” to “Device”, and “Device” to “/dev/zvol/tank/swap”, where “tank” is your ZFS pool name and “swap” is the name of the volume you just created as a swap device.

If you don’t already have ZFS configured, first create your ZFS pool as normal. Then follow the above steps.

Nas4Free allows you to install “embedded” or “full” software installations. I chose “embedded”, because this makes upgrades very easy, and it is simple to install this way on a USB flash device. This means you can dedicate all SATA ports to ZFS storage and not the operating system. With the embedded installation, Nas4Free does not create a swap space. Which means it’s up to the user to do this. It’s possible this isn’t an issue on the “full” installation. I don’t know; I haven’t tried it.

Why is swap necessary? After all, if you have a large amount of RAM, you shouldn’t need swap. And if you use ZFS, you should always have a large amount of RAM. That’s the ideal situation, but in practice it doesn’t work. Why? Because ZFS uses any available memory for cache.

Memory usage with ZFS on FreeBSD.

Memory usage with ZFS on FreeBSD.

Under FreeBSD, this is called wired memory. When ZFS uses wired memory as cache, it should be reclaimed for other uses when it’s needed. That’s certainly what happens under Linux, where disk (read) cache is reported as “Cache memory” in the above graph.

I have found this isn’t the case in practice. Instead, when memory gets tight, FreeBSD kills processes it really shouldn’t:

Apr 30 12:49:45 san1 kernel: pid 69430 (istgt), uid 0, was killed: out of swap space

That’s right. Run ZFS for long enough with no swap, and eventually your iSCSI daemon will be killed. Which means your SAN isn’t accessible any longer. It could also kill the NFS server if you are using the userspace NFS daemon, which is what Nas4Free does.

FreeNAS, from which Nas4Free is forked a similarly-named commercial product, doesn’t have this problem. It defaults to using a certain amount of each disk for swap, which means the average user simply won’t see this problem. This may be a moot point, since the average user in this situation should be a highly-trained engineer. But it happened to me, and I’m documenting it here so it doesn’t happen to you.

Thanks to the fine people of the Nas4Free forums for providing additional information.

Tags: , , , , ,

  1. Victor’s avatar

    Thank you – I’ve been battling the embedded install of Nas4Free the whole evening, and was worried that something else was causing the istgt to be killed. Added 32Gb of swap now – so hope it will behave now. I wonder if there’s a way to prevent OS from killing istgt – seems like it has a strange affection towards istgt.

    Reply

    1. Tyler Wagner’s avatar

      I don’t believe you can protect a process from the OOM killer. It’s a feature of the way shared memory works. Here is an article on how Linux does it. I assume the BSD kernel Nas4Free uses is similar.

      Reply

Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.