And that brings me to a trend I have noticed in the satellite Internet business. There are two camps: radio guys who somehow configure a router, and IP guys who somehow align an antenna. Usually the IP guys do better in business than the radio guys. Perhaps this is because radio guys tend to be former military and unused to competition. Perhaps this is because they think all you have to do is deliver a carrier and everyone is happy. But the customer doesn’t care about Hertz, he cares about bits. And really, the space link is still a Layer 1 problem, and the harder work is above that.

On with revealing a little black magic.

Satellite systems communicate over a variety of frequencies. Modems and other terrestrial equipment commonly communicate between each other in one frequency range, yet the actual spacelink communication happens in another frequency range entirely.

The frequency bands I most commonly deal in my job managing satellite Internet services are (band, receive range, transmit range):

• L-band, Rx 950-1450 MHz, Tx 1250-1750 MHz
• C-band, Rx 3625-4200 MHz, Tx 5850-6425 MHz
• Ku-band, Rx 10.95-12.75 GHz, Tx 13.75-14.5 GHz

There are a lot of ways to get confused here. The definitions of these bands varies from region to region. There are a lot of confusing terms like “standard Ku” and “extended Ku” despite there being overlap between the two ranges they define. The C-band I define above is actually called “extended C-band”, yet there is no “standard C band” as far as I can tell. And the Wikipedia articles on these things are a mess.

Satellite communication is normally a one-way process. Internet communication is by nature two-way, but from a satellite perspective we can regard it as two one-way processes. * So we’ll analyse it from that perspective.

During the entire one-way process, the physical communication signal changes frequencies four times. This is done to avoid radio interference at the satellite and ground antennas, and to avoid electrical resistance when communicating over copper.

Assuming Ku-band communication with the satellite, the devices that do the translating are:

1. BUC, or Block Up-Converter, which steps a carrier up from Tx L-band to Tx Ku-band
2. The frequency translator at the transponder on board the satellite, which steps a carrier down from the Tx Ku-band to the Rx Ku-band for retransmission to Earth
3. LNB, or Low-Noise Block down-converter, which steps a carrier down from Rx Ku-band to Rx L-band

These devices work by using a fixed translation amount, called the LO, for Local Oscillator. If you’ve guessed this is because an actual fixed oscillating clock is used, you are correct! It is very important to know what the LO value is, or you don’t know what frequency your transmission will become.

Let’s say a modem starts transmitting at 1131 MHz over a copper wire. This signal reaches a BUC with an LO of 12800 MHz. 1131 + 12800 = 13931 MHz, which is then transmitted into space as radio. The satellite receives 13931 MHz, where the transponder’s translator has an LO too. Here, it is 1260 MHz, and 13931 - 1260 = 12671 MHz, so 12671 MHz is then transmitted back to Earth as radio. ** Finally this 12671 MHz reaches the customer’s LNB with an LO of 11300 MHz. 12671 - 11300 = 1371 MHz, which is what reaches the customer’s modem.

In summary:

1. 1131 MHz (modem)
2. 1131 + 12800 = 13931 MHz (BUC)
3. 13931 - 1260 = 12671 MHz (transponder translation)
4. 12671 - 11300 = 1371 MHz (LNB)

The process repeats itself when the customer transmits in return, using whatever BUC and LNB are installed in the opposite direction. It’s common for both BUCs to have the same LO, and both LNBs, which makes the math easy for me. Each group of transponders on a satellite has its own transponder translation frequency, and these are different between C-band, Ku-band, K-band, and so on. The average technical user guide for the satellites Talia operates on is 35 pages. Read carefully.

I have a large spreadsheet tracking frequency translations like this. When you call a satellite operator to complain about a problem, they want the downlink frequency (12671 MHz in the example above). They certainly don’t know or care what your L-band frequencies are, because they can be different depending on the LO of your equipment!

A spectrum analyser is a common tool for observing satellite carriers, checking for interference, and measuring transmission power. Spectrum analysers commonly operate in L-band. This means that you must spot a problem in L-band and then do the math to find the downlink frequency before reporting a problem to the satellite operator. It seems to me that internally calculating the translation values would be a simple feature of a spectrum analyser, but most cheap ones (under $10,000 USD) don’t offer this. You’d need to define profiles for your different hardware/satellite configurations, but anyway it’s something that an IP engineer would think to add and a radio engineer just expects they’ll have to do manually.

I have yet to find even a single open source package to manage frequency plans or carrier assignments, by even the simplest methods. My spreadsheet is the most advanced free tool I have for this.

* Sometimes satellite Internet services are one-way. The download link is via satellite, and the upload link via terrestrial modem. But that method is unpopular today.

** I find it interesting that satellites themselves receive in the Tx range and transmit in the Rx range.

In Wordpress 2.6.2, the problem comes from a bug in ./wp-includes/formatting.php. The auto-formatter tries not to do these replacements between code tags, but incorrectly identifies the start of any following tag as the end of the code section. That means a multi-line code section (which has <br> tags breaking each line) will have replacements run on it on all lines after the first.

This bug is found between lines 30 and 43:

if (isset($curl{0}) && '<' != $curl{0} && '[' != $curl{0} && $next && !$has_pre_parent) { // If it's not a tag
   // static strings
   $curl = str_replace($static_characters, $static_replacements, $curl);
   // regular expressions
   $curl = preg_replace($dynamic_characters, $dynamic_replacements, $curl);
} elseif (strpos($curl, '<code') !== false || strpos($curl, '<kbd') !== false || strpos($curl, '<style') !== false || strpos($curl, '<script') !== false) {
   $next = false;
} elseif (strpos($curl, '<pre') !== false) {
   $has_pre_parent = true;
} elseif (strpos($curl, '</pre>') !== false) {
   $has_pre_parent = false;
} else {
   $next = true;
}

The solution is to make code tags behave like pre tags. Change lines 35-39 to the following:

} elseif (strpos($curl, '<kbd') !== false || strpos($curl, '<style') !== false || strpos($curl, '<script') !== false) {
   $next = false;
} elseif (strpos($curl, '<pre') !== false || strpos($curl, '<code') !== false) {
   $has_pre_parent = true;
} elseif (strpos($curl, '</pre>') !== false || strpos($curl, '</code>') !== false) {
   $has_pre_parent = false;

Now code tags will be treated as pre tags - absolutely no replacements except for those necessary for HTML encoding. This does introduce bugs in the case of nested code and pre tags, so don’t put them inside each other.

If you want to disable replacements altogether, just comment out lines 32 and 34 (the str_replace and preg_replace calls).

Aside: writing this article so all the tags would display correctly was an incredible pain.

I’ve packaged the latest release, 0.10.0, for Ubuntu Hardy amd64 and i386. It is available in my repository now. Source packages are provided, so any other Debian-based distro or architecture should be able to build it.

I wrote about something similar on my other blog last year.

First, some background information on VSAT. The single biggest limitation of satellite Internet service is latency. In network terms, latency is the total round trip time for your packet to reach its destination and for the reply to return. When we speak of “ping time”, what we mean is latency.

Here are some common latency measurements in milliseconds (ms):

1 ms - within your LAN
25 ms - my home cable service in London to servers located in mainland UK
90 ms - typical home DSL in the US to google.com
100-150 ms - the transatlantic cable between the UK and New York state
600-2000 ms - typical VSAT remote to hub link

Why is VSAT latency so high? Geosynchronous satellites orbit at an altitude of 35,000 km. VSAT remotes connect to the Internet via a star-shaped network with a hub at the centre. When a remote modem pings google.com, its traffic must travel up to the satellite, then back down to our teleport, and then over the terrestrial Internet to google.com. The reply does the opposite, returning back over the terrestrial Internet, back up to the satellite and down again to the remote. That means every single ping you send is travelling 35,000 x 4 = 120,000 km. Divide by c, and you find the minimum latency for any VSAT is 480 ms. Further time is added due to radio modulation, error correction, router delay, and various technical issues with shared bandwidth satellite networks. All that adds up to a minimum effective latency for all VSAT communications of about 600 ms. If that network is busy, it goes up.

So when one of our customers makes a VoIP phone call, there is an audible (600 ms) delay between when he speaks and when he can expect a reply. We constantly work to keep that number as low as possible. People seem to tolerate up to 250 ms without noticing, but when you start edging up to one second, it quickly becomes intolerable - less like a phone, more like a radio.

This latency is immediately noticeable on ping and traceroute. I prefer mtr, a tool that combines the two. Plain old tracert / traceroute will show you the same results, just a lot slower. Here is an example run from a VSAT link:

mtr yahoo.com

 Host                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.5.1                              0.0%    10    1.3   3.3   1.2   7.2   2.4
 2. 172.16.4.97                              0.0%     9  605.8 672.2 605.8 764.5  54.4
 3. 172.16.4.1                               0.0%     9  699.6 690.7 611.9 761.6  50.7
 4. 172.16.3.1                               0.0%     9  637.1 666.6 578.1 763.7  58.6
 5. kln-145-253-9-73.arcor-ip.net            0.0%     9  671.4 638.1 592.0 699.7  38.2
 6. kln-145-254-9-153.arcor-ip.net           0.0%     9  640.2 638.9 578.5 736.1  46.8
 7. dus-145-254-18-194.arcor-ip.net          0.0%     9  655.3 650.9 609.2 684.5  24.2
 8. amd-145-254-16-130.arcor-ip.net          0.0%     9  702.0 667.6 604.8 778.2  60.2
 9. ge-1-3-0.pat1.ams.yahoo.com              0.0%     9  740.9 665.1 595.4 740.9  47.3
10. ge-1-2-0.pat2.ams.yahoo.com              0.0%     8  677.7 673.2 621.4 739.4  37.2
11. so-3-1-0-pat1.the.yahoo.com              0.0%     8  711.9 680.5 616.7 773.4  56.4
12. so-2-1-0-pat1.nyc.yahoo.com              0.0%     8  706.9 706.0 682.0 732.4  17.1
13. so-3-0-0.pat1.dcp.yahoo.com              0.0%     8  685.3 726.9 685.3 755.4  29.3
14. ae2-p170.msr2.re1.yahoo.com              0.0%     8  731.8 747.0 731.3 779.9  18.3
    ae2-p160.msr1.re1.yahoo.com
15. ge-9-3.bas-a1.re4.yahoo.com              0.0%     8  677.9 757.3 677.9 819.1  45.2
    te-8-3.bas-a2.re4.yahoo.com
16. w2.rc.vip.re4.yahoo.com                  0.0%     8  789.1 785.1 708.1 853.4  48.6

Note the huge jump in latency between hops 1 and 2. This must be the space link - hop 1 is the remote modem and hop 2 is the hub modem (the one in the teleport).

The jump in latency is obvious in both directions. From a terrestrial link to a VSAT IP:

 Host                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1                              0.0%    25    0.9   1.1   0.6   6.2   1.1
 2. 192.168.55.34                            0.0%    25    1.1   0.7   0.4   1.1   0.2
 3. 172.16.21.71                             0.0%    25    4.5   4.0   3.6   4.9   0.3
 4. 217.239.41.18                            0.0%    25   39.5  41.5  39.1  67.7   6.9
 5. 193.159.224.254                          0.0%    25   44.6  88.3  44.1 596.8 136.2
 6. 83.137.59.9                              0.0%    24   41.7  42.3  41.4  49.5   1.6
 7. 192.168.140.4                            0.0%    24   42.5  42.7  41.6  44.6   0.6
 8. 192.168.100.50                           0.0%    24   48.1  47.4  46.2  50.3   0.9
 9. 192.168.103.149                          0.0%    24  613.7 719.6 592.2 1128. 127.2
10. 80.86.24.6                               0.0%    24  620.9 787.1 582.9 1678. 272.7

Here the space link is between hops 8 and 9.

Similar increases in latency are detectable along undersea fibre. From my home in London to my alma mater Cal Poly:

 1. gw-belafonte                             0.0%     5    1.0   1.1   1.0   1.2   0.1
 2. 10.181.120.1                             0.0%     5   56.3  41.1  23.5  56.3  14.6
 3. osr01camd-v15.network.virginmedia.net    0.0%     5   32.1  39.9  19.0  89.7  28.8
 4. osr02hari-tenge71.network.virginmedia.n  0.0%     5   40.6  34.6  15.3  54.2  15.0
 5. pop-bb-b-ge-200-0.network.virginmedia.n 20.0%     5   46.1  34.6  18.6  46.1  13.2
 6. gfd-bb-a-so-010-0.network.virginmedia.n 20.0%     5   42.5  47.7  41.7  54.1   6.5
 7. 213.152.245.49                           0.0%     5   23.8  45.8  23.8  65.3  19.1
 8. so-0-0-0.mpr2.ams5.nl.above.net          0.0%     5   46.0  58.0  35.2  83.0  18.3
 9. so-2-0-0.mpr1.lhr2.uk.above.net          0.0%     5   65.2  74.8  47.3 100.2  19.8
10. so-1-1-0.mpr1.dca2.us.above.net          0.0%     5  124.5 120.6 105.2 135.4  10.9
11. so-1-2-0.mpr1.lga5.us.above.net          0.0%     5  132.6 152.9 121.6 216.0  39.1
12. so-2-1-0.mpr1.sjc2.above.net             0.0%     5  187.3 205.8 183.9 251.1  27.0
13. so-4-0-0.mpr3.pao1.us.above.net          0.0%     5  280.9 227.7 200.4 280.9  33.3
14. paix-px1--abovenet.cenic.net            20.0%     5  267.4 232.9 189.2 279.1  46.9
15. dc-svl-core1--svl-dc1-ge.cenic.net       0.0%     5  255.3 240.6 206.3 274.7  26.5
16. dc-sol-agg1--svl-core1-ge.cenic.net      0.0%     5  218.7 216.9 196.6 249.3  19.8
17. dc-sol-agg2--sol-agg1-ge-2.cenic.net    25.0%     5  209.8 209.3 199.1 218.9   9.9
18. dc-slo-dc1--sol-dc2-pos.cenic.net        0.0%     5  240.6 221.6 203.3 240.6  19.9
19. primary-pix-outside.netadm.calpoly.edu  25.0%     5  271.3 247.0 200.7 271.3  40.1
20. 129.65.1.41                              0.0%     5  245.9 213.7 190.7 245.9  23.3
21. ???

Now we can see that my home is named “Belafonte” (hop 1), that I have a network connection via Virgin Media (hops 3-6), that Virgin Media finds it cheaper to route via Above.net through Amsterdam and back to the UK rather than direct within the UK (hops 8 and 9), and that we cross the Atlantic between hops 9 and 10. Also, Cal Poly should know better than to use DNS names that state both function and operating system to anyone looking (hop 19, a Cisco PIX firewall at their network’s border).

Note the 60 ms increase in crossing the Atlantic, which is actually quite good for such a long fibre run. Compare a route from Dubai to the UK:

 1. 192.168.1.1                              0.0%     7    0.5   0.8   0.5   2.4   0.7
 2. 195.229.244.26                           0.0%     7   10.9  18.4   8.0  48.8  14.9
 3. 195.229.245.98                           0.0%     7   15.3  17.0   7.5  28.3   9.4
 4. 194.170.0.234                            0.0%     7    7.5   8.3   7.5   8.8   0.5
 5. 195.229.1.181                            0.0%     7   10.8  14.4  10.8  28.7   6.6
 6. 195.229.1.166                            0.0%     7   14.4  14.8  14.1  15.4   0.5
 7. pos10-0.cr02.hkg05.pccwbtn.net           0.0%     7  292.6 293.7 292.3 298.9   2.6
 8. uunet.pos1-2.cr02.frf02.pccwbtn.net      0.0%     7  258.8 259.1 258.8 259.5   0.3
 9. ge-1-2-0.TL2.FFT1.ALTER.NET              0.0%     6  259.1 259.1 258.6 259.6   0.3
10. so-0-0-0.XR2.LND9.ALTER.NET              0.0%     6  247.2 250.8 246.7 266.3   7.8
11. POS2-0.GW9.LND10.ALTER.NET               0.0%     6  258.3 259.7 258.3 262.4   1.9

Care to spot the fibre there? Here’s the kicker - it looks like it’s routing direct from the UAE to the US, then on to the UK! If memory serves, that’s probably the FLAG cable that runs from the Persian Gulf to the West Coast of the US. Yes, they make runs that long!

1. Allow root to display applications on the user’s desktop

Normally only the logged-in user has the authority to run X applications. This is a good thing, and a useful security feature. Running commands as your user with sudo allows that application to use your X session, like so:

sudo wireshark

However, this doesn’t work:

sudo -i
(now as root) wireshark

We can work around this by modifying root’s ~/.Xauthority file. Either copy yours or link it into /root/, and you’re done:

cd /root/
ln -f -s /home/username/.Xauthority

If you have multiple users on the same machine, you can achieve the same goal by merging several files using the xauth command.

2. Retain environment variables when you execute sudo

OK, now root can make X apps display on tyler’s display, but he still has to set the DISPLAY variable or pass a -display argument. So let’s keep this variable when we login as root with “sudo -i“.

Add the following somewhere in /etc/sudoers:

Defaults                 env_keep+="DISPLAY XAUTHORITY"

Or just for one user:

Defaults:username        env_keep+="DISPLAY XAUTHORITY"

Now DISPLAY, and if you use it, XAUTHORITY, follow you if you use sudo. You can add any others to this list if you like.

3. Keep the same GTK preferences as root

Now, your root GTK apps run and display, but they look like something from Windows 95. This is because GTK has absolutely horrific default settings which are nearly always overridden by your window manager’s theme, and root doesn’t have your GTK preferences. That’s easy to fix:

cd /root/
ln -f -s /home/username/.gtkrc-2.0

And if you use KDE with the “Use my KDE style in GTK applications” (gtk-qt-engine) setting, you might also want:

cd /root/
ln -f -s /home/username/.gtk_qt_engine_rc

Finally, I just like a better sudo prompt. So I set this at the botton of /etc/bash.bashrc on every machine I manage:

export SUDO_PROMPT="[sudo] password for %u@%h: "

This is better than the sudo default (the totally ambiguous “Password: “) or the Ubuntu default (”[sudo] password for username: “) because it also lists the host you are currently on. If you login to as many machines as I do, confusing one sudo prompt for another is an easy way to ruin your day.

We start with an install on a single 80 GB SATA drive, partitioned as follows:

/dev/sda1 as /, 10GB, ext3, bootable
/dev/sda2 as swap, 1GB
/dev/sda3 as /home, 69GB, ext3

We want to add a second 80GB SATA drive and move the entire install to use RAID1 between the two drives. So the final configuration will appear:

/dev/md0 as /, 10GB, ext3
/dev/md1 as swap, 1GB
/dev/md2 as /home, 69GB, ext3

Where the raid arrays are:

md0 : active raid1 sda1[0] sdb1[1]
md1 : active raid1 sda2[0] sdb2[1]
md2 : active raid1 sda3[0] sdb3[1]

Here there be dragons. As always, back up your data first. If you don’t know how to use rsync, now is an excellent time to learn.

The general plan is:

1. Partition the new drive
2. Create RAID arrays on the new drive
3. Copy the data from drive 1 to the new RAID arrays on drive 2
4. Install grub on drive 2
5. Configure fstab and mdadm.conf, and rebuild initramfs images
6. Reboot on the RAID arrays on drive 2
7. Repartition drive 1 and add it to RAID

All commands are run as root. Use sudo if you prefer.

Step 1: Partition the new drive

Assuming you want to partition the second drive the same way as the first, this is easy. Just clone the partitions from /dev/sda to /dev/sdb:

sfdisk -d /dev/sda | sfdisk /dev/sdb

Then use parted to mark them for software RAID, and the first partition to boot:

parted /dev/sdb
(parted) toggle 1 raid
(parted) toggle 2 raid
(parted) toggle 3 raid
(parted) toggle 1 boot
(parted) print

Disk /dev/sdb: 80.0GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      0.51kB  10.0GB  10.0GB  primary  ext3         boot, raid
 2      10.0GB  11.0GB  2000MB  primary  linux-swap   raid
 3      11.0GB  80.0GB  69.0GB  primary  ext3         raid
(parted) quit

Parted will show you a file system in each partition, but the reality is that they will be plain linux software raid partitions (type 0xfd):

fdisk -l /dev/sdb

Disk /dev/sdb: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xa3181e57

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1        1216     9765625   fd  Linux RAID autodetect
/dev/sdb2            1216        1338      976562+  fd  Linux RAID autodetect
/dev/sdb3            1338        9730    67408556   fd  Linux RAID autodetect

Step 2: Create RAID arrays on the new drive

Now we create RAID 1 arrays on each partition. These arrays will have just one member each when we create them, which isn’t normal for RAID 1. So we’ll have to force mdadm to let us:

apt-get install mdadm
mdadm --create /dev/md0 --level=1 --force --raid-devices=1 /dev/sdb1
mdadm --create /dev/md1 --level=1 --force --raid-devices=1 /dev/sdb2
mdadm --create /dev/md2 --level=1 --force --raid-devices=1 /dev/sdb3

Step 3: Copy the data from drive 1 to the new raid arrays on drive 2

This is a job for rsync. First, there are some directories on a running Linux system that we do not want to copy, like /dev and /proc. We also want to ignore tempfs directories, like /var/run. The best way to avoid these is to make an excludes file. Create a file, /root/rsync-filter, with the following content:

- /dev/
- /lib/modules/*/volatile/
- /mnt/
- /proc/
- /sys/
- /var/lock/
- /var/run/

These lines define directories we will not copy over. You may wish to add /tmp, apt’s cache, etc, but if you do you must manually create the directories themselves on the new filesystem.

Mount the new RAID array:

mount /dev/md0 /mnt
mkdir /mnt/home
mount /dev/md2 /mnt/home

If you are using a different mount structure, just be sure to recreate it and mount it in the right places in the new filesystem under /mnt/.

And start the rsync copy:

rsync -avx --delete -n --exclude-from /root/rsync-filter / /mnt/
rsync -avx --delete -n /home/ /mnt/home/

You will see a list of files that will be changed, but nothing actually happens. This is the job of the -n argument, which performs a dry-run. Always do this before actually starting a copy. You WILL make a painful mistake with rsync some day, so learn to be cautious. Repeat the above commands without the -n when you are sure all is well.

The -x argument ensures that you will not cross filesystem boundaries, which is why you must copy /home separately, and any other mounted filesystems. If you omit this you only need one command. But make sure you have a good rsync filter file, and that you have nothing mounted like /media/cdrom that you don’t want an archive of.

Finally, create the directories that you skipped with your filter:

cd /mnt/
mkdir -p dev/ mnt/ proc/ sys/ var/lock/ var/run/
for i in /lib/modules/*/volatile ; do mkdir -p /mnt/$i ; done

Step 4: Install grub on drive 2

OK, we almost have a working RAID install on the second drive. But it won’t boot yet. Let’s use chroot to switch into it.

mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
chroot /mnt

Now we have a working /dev and /proc inside the new RAID array, and by using chroot we are effectively in the root of the new array. Be absolutely sure you are in the chroot, and not the real root of drive 1. Here’s an easy trick: make sure nothing is in /sys:

ls /sys

If not, you’re in the new chroot.

Now we just need to edit grub’s device.map and menu.list files. Edit /boot/grub/device.map and make sure both drives are listed:

(hd0)   /dev/sda
(hd1)   /dev/sdb

Now get the UUID of /dev/md0’s filesystem:

tune2fs -l /dev/md0 | grep UUID

Filesystem UUID:          8f4fe480-c7ab-404e-ade8-2012333855a6

Edit /boot/grub/menu.list and find the line like this:

# kopt=root=UUID=9e299378-de65-459e-b8b5-036637b7ba93 ro

Replace the UUID with the one you just found. Leave the line commented, and save the file. Now rebuild menu.lst to use the new UUID:

update-grub

Double-check that each boot option at the bottom of menu.lst is using the right UUID. If not, edit them too. Finally, install grub on the second drive.

grub-install /dev/sdb

Step 5: Configure fstab and mdadm.conf, and rebuild initramfs images

We’re almost ready to reboot. But first, we need to build an initramfs that is capable of booting from a RAID array. Otherwise your boot process will hang at mounting the root partition. Still in the chroot, edit /etc/fstab and change the partition entries to the new filesystems or devices.

If your /etc/fstab has “UUID=” entries like the following, change them to the new entries:

proc            /proc           proc    defaults        0       0
UUID=8f4fe480-c7ab-404e-ade8-2012333855a6 /               ext3    relatime,errors=remount-ro 0       1
UUID=c1255394-4c42-430a-b9cc-aaddfb024445 none            swap    sw              0       0
UUID=94f0d5db-c525-4f52-bdce-1f93652bc0b1 /var/           ext3    relatime 0       1
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

In the above example, the first UUID cooresponds to /dev/md0, the second to /dev/md1, and so on. Find the UUIDs with:

/lib/udev/vol_id /dev/md0

If your /etc/fstab has “/dev/sda1″ entries like it’s a bit easier. Just change them to /dev/md0 and so on:

proc            /proc           proc    defaults        0       0
/dev/md0        /               ext3    relatime,errors=remount-ro 0     1
/dev/md1        none            swap    sw              0       0
/dev/md2        /var/           ext3    relatime        0       1
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

Now, while still in the chroot, edit /etc/mdadm/mdadm.conf to list each RAID array:

DEVICE partitions
ARRAY /dev/md0 UUID=126a552f:57b18c5b:65241b86:4f9faf62
ARRAY /dev/md1 UUID=ca493308:7b075f97:08347266:5b696c99
ARRAY /dev/md2 UUID=a983a59d:181b32c1:bcbb2b25:39e64cfd
MAILADDR root

Find the UUID of each RAID array, which is not the same as the UUID of the filesystem on it (!), using mdadm:

mdadm --detail /dev/md0 | grep UUID

Now, rebuild your current kernel’s initramfs image:

update-initramfs -u

Or all of them:

update-initramfs -u -k all

Step 6: Reboot on the RAID arrays on drive 2

Now we’re ready to reboot. First, exit the chroot and power off the machine cleanly. You have three options:

1. If your BIOS allows you to select which drive to boot from, elect to boot from drive 2.
2. Swap drives 1 and 2 so drive 2 becomes /dev/sda, and restart
3. Use a USB recovery stick to boot from drive 2

When the system restarts, you should reboot on the new RAID drive. Make sure:

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              9.2G  1.9G  6.9G  21% /
varrun                498M   56K  498M   1% /var/run
varlock               498M     0  498M   0% /var/lock
udev                  498M   68K  498M   1% /dev
devshm                498M     0  498M   0% /dev/shm
lrm                   498M   39M  459M   8% /lib/modules/2.6.24-19-generic/volatile
/dev/md2               63G  130M   60G   1% /home

Step 7: Repartition drive 1 and add it to RAID

Finally, we add the old drive into the array. Assuming you chose option 2 and didn’t swap the drive’s cables:

sfdisk -d /dev/sdb | sfdisk --force /dev/sda

If you receive a warning message here, reboot now.

Re-reading the partition table ...
BLKRRPART: Device or resource busy
The command to re-read the partition table failed
Reboot your system now, before using mkfs

If not, continue on:

mdadm /dev/md0 --grow -n 2
mdadm /dev/md1 --grow -n 2
mdadm /dev/md0 --add /dev/sdb1
mdadm /dev/md1 --add /dev/sdb2

If you chose option 1, just swap “/dev/sda” and “/dev/sdb” everywhere above.

The RAID array will now rebuild. To check its progress:

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
      497856 blocks [2/2] [UU]

md1 : active raid1 sdb2[2] sda2[0]
      3696192 blocks [2/1] [U_]
      [>....................]  recovery =  0.4% (16320/3696192) finish=7.4min speed=8160K/sec

unused devices:

Once the array is finished rebuilding, reinstall grub on the new drive. We’ll do both, for good measure.

grub-install /dev/sda
grub-install /dev/sdb

You should now be able to reboot without either drive, and your system will come up cleanly. If you ever need to replace a failed drive, remove it, use step 7 above to clone the partition scheme to the new drive and add it to the array.

Gotchas:

If you are having issues with your RAID setup, especially if you have an older RAID setup or older release already, you might need these packages:

apt-get install lvm2 evms dmsetup

One thing I’ve noticed about Linux, even desktop-focused distributions, is that it’s largely tuned to the most desirable behaviour for a server. But what is desirable on a server isn’t necessarily desirable on a desktop. On Ubuntu Linux (or any Debian-based distro), add the following two lines to /etc/sysctl.conf:

vm.swappiness=1
vm.vfs_cache_pressure=50

These changes will be done automatically on reboot, but let’s load them now with:

sysctl -p

Now you’ll notice GUI applications feel more responsive and filesystem browsing will be faster when the disk is busy. What we have done is instruct the kernel to prefer to keep applications in RAM (swappiness) instead of disk cache, and to prefer to keep filesystem inodes and directory entries cached at the cost of files themselves. This means reading files from disk is slightly slower (IE, copying, moving, loading big files), but browsing the filesystem will be much faster.

Rudd-o.com wrote extensively on this last year. I recommend his write-up for anyone who wants to understand the issues in more depth.

For the record, I ran the tests in Rudd-o’s article. He advised creating a big file, then displaying the entire filesystem hierachy with find, and then copying that file. In my case, I went from 2 minutes, 6 seconds for the “find / > /dev/null” to just 17 seconds. That has made browsing in Konqueror a great deal faster.