Moving cPanel services and /home/virtfs

We are currently moving our customer hosting platform from our old colo to our new one. The servers run cPanel on CentOS 4. The process of moving these services goes something like this:

1. install new servers at new location of same or similar model as the old servers, and load them with a basic CentOS install
2. rsync oldserver:/ to newserver:/ including customer files
3. reboot new server and fix whatever goes wrong (MAC addresses changing, etc)
4. move BGP announcement of hosting IP range to new colo
5. shut down and remove old servers

The actual process is more complicated. During step 2, I hit a problem with cPanel’s /home/virtfs directory tree.

cPanel users with shell access use a facility called jailshell. Jailshell limits a user to a subtree of the filesystem that only reveals their own files. Jailshell does this by creating a basic root directory structure under /home/virtfs/username, and using rbind mounts to give access to parts of the root filesystem.

This has been documented elsewhere, but you should not run the commands in that blog. Thankfully the author made a syntax error in his “for .. umount” script, because if it did run correctly you’d unmount all your filesystems including /.

These files in /home/virtfs are NOT hardlinks or symlinks, but are actually the real files in your root filesystem. If you delete them, you’ll delete the real files in /. Mount’s “rbind” feature allows one directory to be mounted in several places in the filesystem, which is what is happening here.

These directories should be mostly empty when users aren’t logged in. Sometimes users fail to logout from jailshell, and in that case the rbind mounts will remain. You can remove them, but only after you unmount the rbind mounts. First make sure no users are logged in:

root@example.com:~# who
root     pts/0        Oct 28 06:13 (10.0.0.1)

Then you can unmount these directories with a loop:

for i in `cat /proc/mounts | grep /home/virtfs | cut -d ' ' -f 2 ` ; do umount $i ; done

My concern is using rsync to clone my server. When I do this, I do not want to copy /home/virtfs at all. So I’ll use an rsync excludes file. First create /root/rsync-filter:

- core.*
- /dev/
- /lib/modules/*/volatile/
- /media/
- /mnt/
- /proc/
- /sys/
- /tmp/
- /var/lock/
- /var/run/
- /var/tmp/
- /boot/grub/
- /etc/mdadm/
- /etc/fstab
- /etc/mtab
- /etc/resolv.conf
#- /etc/sysconfig/network-scripts/ifcfg-eth0*
- /home/virtfs/
- /root/screenlog.*
- /root/rsync-filter
- /var/spool/cron/root

The first part of this is my “boilerplate” rsync exclude that I always use when cloning servers. If you use ‘rsync -x’ you can leave some of it out, but it’s best to be safe. Note that I’ve commented out the line for ifcfg-eth0. This is because, while I’m testing, I want my new server to stay on a temporary IP address. Once I clone that file it will come up on the old server’s IP, so I’ll only do that last, just before the final reboot.

You can also see that I’m skipping all of /home/virtfs. Since the contents here are created on demand by jailshell, you won’t need it.

I’m ready to clone the server, so I run (on the old server):

rsync -av --delete --exclude-from /root/rsync-filter / root@newserver:/ -n

Once you verify that is doing exactly what you expect, remove the ‘-n’ and run it again. If you are in the habit of running rsync without a dry-run first, you will eventually regret it. Like C, rsync gives you enough rope to shoot yourself in the foot.

The rest of the migration is another story.