OpenWRT 10.03 on Buffalo WZR-HP-G300NH

I’ve been using a Linksys WRT54GL router running DD-WRT for several years now. It runs OpenVPN and fits a surprisingly large number of features into 4 MB of flash and 16 MB of RAM. It has been stable and reliable, but it’s a very under-powered device by modern standards. I haven’t been happy with the pace of DD-WRT development, which takes a very long time to release large, monolithic updates. First I tried to upgrade it to OpenWRT, but the latest release is too large to incorporate OpenVPN in 4 MB of flash. So I replaced it with a Buffalo WZR-HP-G300NH running OpenWRT Backfire 10.03.

Full disclosure: before replacing the WRT54GL router, I tried to downgrade it to an older release of OpenWRT (Kamikaze 8.09.2). I uploaded the wrong firmware and bricked it. Oops. I had already decided to replace it, but this added some urgency.

My reasons for upgrading are mostly technical. I want to install other software packages on the router, such as mtr and tcpdump. This makes debugging infinitely faster and easier. When the going gets tough, the tough sniff packets. The Buffalo also represents a hardware upgrade in virtually every way – flash storage, RAM, CPU, wireless speed, Ethernet speed, and USB support.

However, there are also a bit of politics involved. DD-WRT’s development model is much more “closed” than OpenWRT, and is heavily centralised. OpenWRT is more of a community project and supports external packages in parallel development. Further, DD-WRT tries to store everything in the device NVRAM (non-volatile RAM), while OpenWRT creates a JFFS2 filesystem much more like a typical Linux system. I’d rather use and support OpenWRT as a project, and I think their way is the way forward.

Installation

The OpenWRT page for the WZR-HP-G300NH gives step-by-step instructions on how to install OpenWRT. These steps worked perfectly, so I won’t reproduce them here. I installed the Squashfs image via the TFTP method. After installation, I logged in via telnet and changed the root password.

There is a known issue where SSH doesn’t start for 2-3 minutes after you change the root password. This happened to me but was just a matter of waiting until SSH started.

The hardware takes about 20 seconds to boot up, then waits for firmware by TFTP for 4 seconds, then loads OpenWRT. In total, boot time is about 45 seconds. This is just longer than my Ubuntu 10.04 media PC waits for an address by DHCP, so whenever I reboot the router I have to manually reconnect to the network. Ironically, the PC boots faster than the router. I hope makers of embedded systems follow Ubuntu’s example of speeding up boot times, because the difference is pretty glaring.

Package Management

Packages can be installed from the command-line or from Luci, the web interface. Before you can do so, you must download a package list from OpenWRT using the package manager, opkg. This tool is similar to dpkg or apt-get from Debian. The package list is kept in a RAM disk, so it does not persist across reboots. Before installing, either run opkg update or login to Luci and click “Administration”, then “System -> Software”, and finally “update package list”.

Install packages with opkg install packagename. Some packages, such as dropbear and openssh-client may overwrite the same files, so you must use the --force-overwrite switch if you see something like this:

root@gw-belafonte:/# opkg install openssh-client
Installing openssh-client (5.4p1-1) to root...
Downloading http://downloads.openwrt.org/backfire/10.03/ar71xx/packages/openssh-client_5.4p1-1_ar71xx.ipk.
Collected errors:
 * check_data_file_clashes: Package openssh-client wants to install file /usr/bin/scp
        But that file is already provided by package  * check_data_file_clashes: dropbear
 * check_data_file_clashes: Package openssh-client wants to install file /usr/bin/ssh
        But that file is already provided by package  * check_data_file_clashes: dropbear
 * opkg_install_cmd: Cannot install package openssh-client.

Solution:

opkg --force-overwrite install openssh-client

Most packages are configured by the uci command or Luci, and store their preferences in /etc/config.

Remove packages with opkg remove packagename. This does not leave configuration files behind, except for those in /etc/config.

I must say I’m pretty impressed with opkg; it’s an excellent implementation of principles from apt-get and it works very well on such a small platform. It’s certainly much easier to use than DD-WRT, which doesn’t have packages at all. You must select what features you want when you install the firmware, and the choices are slim – standard, micro (small flash), and openvpn are about the only useful choices. It is possible to install OpenWRT packages on DD-WRT devices with sufficient flash, but then why not just use OpenWRT?

Daemons like OpenVPN do not start when they are installed, and they are not configured to start on boot. To do this, either run:

/etc/init.d/openvpn enable
/etc/init.d/openvpn start

Or use Luci. Go to “Administration”, then “Services -> Initscripts”, and click on both “Enable” and “Start”.

Wireless

Wireless doesn’t work out of the box, but it is easy to install. If you want to use WEP or WPA, install “wpad-mini”. If you want a more advanced mechanism like RADIUS or EAP, install the full “wpad” package. I preferred to install “wpad” rather than “wpad-mini”, so I could support other WPA authentication mechanisms. We’re always talking about replacing pre-shared keys with RADIUS at work, and the difference in package size is minimal, so I installed the full package.

opkg update
opkg install kmod-ath9k wpad
rm -f /etc/config/wireless
wifi detect > /etc/config/wireless

After that I was able to configure wifi using Luci. You can configure multiple wifi networks but I found the ath9k driver (or chipset) didn’t support different encryption settings. For instance, you can have these:

1. One network with WPA encryption
2. One network with WPA and one network with no encryption

But you cannot have these:

1. One network with WPA and one network with WEP
2. Two WPA networks with different keys

I’ve read that it is possible to do this with software encryption, but it wasn’t sufficiently important to me to continue. If I really want to use my Nintendo DS, I’ll just crack a nearby WEP network or set up a temporary AP on spare hardware I have lying around.

I didn’t test 802.11n wireless, as I have no other hardware which supports it.

USB

The instructions for enabling USB support said to only install the packages you really need. However, the WZR-HP-G300NH has plenty of flash, and I found my FAT32 USB stick wouldn’t mount with just the UTF8 character set installed. So I installed support for all languages and filesystems.

opkg install kmod-fs-btrfs kmod-fs-ext2 kmod-fs-ext3 kmod-fs-ext4 kmod-fs-isofs kmod-fs-reiserfs kmod-fs-vfat kmod-fs-xfs
opkg install kmod-nls-cp1250 kmod-nls-cp1251 kmod-nls-cp437 kmod-nls-cp775 kmod-nls-cp850 kmod-nls-cp852 kmod-nls-cp866 kmod-nls-iso8859-1 kmod-nls-iso8859-13 kmod-nls-iso8859-15 kmod-nls-iso8859-2 kmod-nls-koi8r kmod-nls-utf8
opkg install kmod-usb2 kmod-usb-storage kmod-usb-storage-extras block-hotplug block-mount hotplug2

I then configured the USB drive to mount automatically.

mkdir /mnt/usbstorage -p
uci set fstab.@mount[0].target=/mnt/usbstorage
uci set fstab.@mount[0].device=/dev/sda1
uci set fstab.@mount[0].fstype=auto
uci set fstab.@mount[0].enabled=1
uci set fstab.@mount[0].options=rw,sync,noatime,nodiratime
uci commit fstab
/etc/init.d/fstab enable
/etc/init.d/fstab restart

I didn’t install Samba support. SSH/SCP/SSHFS works fine for me.

HTTPS

The default installation supports SSH but not HTTPS. This strikes me as a strange choice, but it’s easy to resolve:

opkg install luci-ssl

Luci didn’t answer HTTPS until I rebooted the router.

SNMP

I like Cacti, and I like graphing my router’s IP throughput. OpenWRT has two SNMP daemons: mini_snmpd and snmpd. Don’t use mini_snmpd. It only supports SNMP 1/2, and it doesn’t care about SNMP community – it gives SNMP data to anybody who asks for it. It also only runs if you install IPv6 support (package kmod-ipv6).

opkg install snmpd

SNMPd doesn’t have a Luci interface, but it is easy to configure from SSH. Login, and edit /etc/config/snmpd. Change the public and private communities to something less obvious, such as a random password generated by pwgen (also installable with opkg).

config com2sec public
        option secname ro
        option source default
        option community aikeev4H

config com2sec private
        option secname rw
        option source localhost
        option community Heev7nai9aeg4foo3uN7otaing

The private (write) community cannot do much to this device, but I prefer to set it to something very long and then forget about it. Then commit the change and start snmpd (either at command line or Luci’s Initscripts interface).

uci commit snmpd
/etc/init.d/snmpd enable
/etc/init.d/snmpd start

UPNP, Avahi, and NTP Client

I also installed UPNP, Avahi (MDNS), and NTP client support. UPNP and the NTP client can be configured through Luci, but Avahi doesn’t really have anything to configure. OpenWRT supports rdate out of the box, but I use NTP everywhere else, so why not here?

opkg install avahi-daemon luci-app-ntpc luci-app-upnp miniupnpd ntpclient

Enable them using the Luci Initscripts interface.

Utilities

Then I installed all the useful command-line apps that I like to use on routers.

opkg install --force-overwrite bwm conntrack-tools fdisk fping iftop ip iptables-utils lft lsof mtr net-tools-hostname ngrep nmap openssh-client rsync screen snmpd sshfs tcpdump vim

openssh-client is necessary for SSH agent support, which Dropbear’s own SSH client doesn’t offer. I use this with SSHFS to mount my harddrive and dump tcpdump files over the network. This is dangerous, since tcpdump might sniff the SSHFS traffic. Always remember to exclude your own SSH traffic, or bad things will happen:

tcpdump -i eth0 'not tcp port 22'

iptables-utils provides iptables-save, which is a far more sane way to read iptables rules. I can’t even look at normal iptables output any more. fping is ping’s smarter, older, multi-tasking brother. In fact, just try and learn to use all of these utilities. You won’t regret it.

OpenVPN

OpenVPN installation is as easy as the rest:

opkg install openvpn luci-app-openvpn

Next, configure OpenVPN using Luci. It will appear under “Administration”, then “Services -> OpenVPN”. I have an OpenVPN server at work we wish to connect to, so I created a new configuration named “Work”, and then added entries for “Remote host” and “Type of used device” (we use tap, not tun). I then added “Certificate authority”, “Local certificate”, and “Local private key” and uploaded each of the files. I then started OpenVPN using the Initscripts interface.

OpenWRT uses firewall “zones” like “lan” and “wan” to distinguish between outside and inside traffic. Traffic from the “wan” zone which is unknown will be rejected, and traffic from the “lan” group will be masqueraded (NATed) and permitted to exit. This is an excellent design, but it doesn’t have an entry for “vpn”. I tried to create one using Luci (“Administration”, then “Network -> Firewall”), but it doesn’t really work. This is because the firewall loads before OpenVPN. Therefore the tap/tun device does not yet exist, and no rules are added to the firewall to identify the OpenVPN traffic as part of the “vpn” zone. I tried reloading the firewall after OpenVPN, but this caused the WAN interface and VPN to drop, and never worked well no matter how I tried it.

I was really keen on using the Luci interface to configure the firewall, but in the end it was much easier to hack up a solution. I removed my “vpn” zone and just added some lines to /etc/firewall.user. This is a handy hook the developers added, a place to run custom commands after the firewall loads. Here is mine:

root@gw-belafonte:~# cat /etc/firewall.user
# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# 2010-08-26 tyler - OpenVPN
iptables -I INPUT -i tap+ -j ACCEPT
iptables -I FORWARD -i tap+ -j ACCEPT
iptables -I FORWARD -o tap+ -j ACCEPT
iptables -t nat -A POSTROUTING -o tap+ -j MASQUERADE

This means all traffic to/from VPN interfaces is accepted (“tap+” matches tap0, tap1, etc), and traffic to the VPN is masqueraded. A quick reboot, and the VPN started up, the firewall rules loaded correctly, and everything worked perfectly.

LED Configuration

This part is just plain fun. Using Luci (“Administration”, then “System -> LED Configuration”) you can control the behaviour of the LEDs on the front of the device. They can be set to stay on or off, can blink with activity, and can be tied to any network device. You can even set one to heartbeat at a rate tied to uptime load. On the WZR-HP-G300NH you can control these LEDs on the face: orange “security”, green “wireless” and “router”, red “diag”; and you can also control the blue “USB” light on the back just above the USB port. I was not able to make the USB LED turn on when a USB stick is mounted, but I was able to configure the first three lights on the face, so that I am alerted to activity on the LAN, WAN, and VPN interfaces. It’s nice to have a “lock” icon when the VPN is connected, as this is a useful external indicator of network trouble.

Syslog

OpenWRT supports logging to an external syslog server. Using Luci, see “Administration”, “System -> System”, and add additional field “External system log server”. Local logs are stored in a circular buffer, and are visible in Luci (“Administration”, then “Status -> System Log”) and from the command line with logread -f. This is very helpful for debugging OpenVPN problems like bad certificates and expired certificate due to a bad system time.

QoS

Quality of Service (QoS, or rate shaping) is done by the qos-scripts package:

opkg install qos-scripts

To configure it, edit /etc/config/qos and set your upload and download rates, and set rules to classify traffic. Then enable and start it using the Luci Initscripts interface. I have a fairly high-bandwidth connection at home and don’t really need QoS, so I don’t use it.

Performance Metrics and Comparison

Total remaining flash storage after all these packages are installed is 17.3 MB. This means the entire system plus my extras are installed in under half the available flash storage. RAM usage just after reboot, while running openvpn, miniupnp, avahi, dnsmasq, ntpclient, dropbear, uhttpd (Luci) with SSL support, and snmpd is 34 MB.

root@gw-belafonte:/etc/config# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.8M      2.8M         0 100% /rom
tmpfs                    30.2M     88.0K     30.2M   0% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock4           27.6M     10.3M     17.4M  37% /overlay
mini_fo:/overlay          2.8M      2.8M         0 100% /

root@gw-belafonte:/etc/config# free
              total         used         free       shared      buffers
  Mem:        61944        34208        27736            0         1864
 Swap:            0            0            0
Total:        61944        34208        27736

So far it has handled Bittorrent without complaint. Kernel netfilter settings seem to be a fair balance of permissive and careful; TCP timeouts are one hour, UDP timeouts are 60 and 180 seconds, and conntrack can handle up to 16K connections. At 232 bytes each, netfilter connection tracking will consume just under 4 MB of RAM. This is significantly more than the Linksys could support. Edited for brevity:

root@gw-belafonte:/etc/config# sysctl -a | grep tcp_timeout
net.netfilter.nf_conntrack_tcp_timeout_established = 3600

root@gw-belafonte:/etc/config# sysctl -a | grep udp_timeout
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 180

root@gw-belafonte:/etc/config# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
16384

I compared wifi throughput between a Linksys WRT54GL and the new Buffalo WZR-HP-G300NH. Thankfully I had a spare unit to replace the bricked WRT54GL. Tests were performed using iperf, transmitting UDP datagrams for 30 seconds at various speeds, and monitoring the test with bwm-ng. Data was sent between my gigabit-capable media PC on Ethernet and my laptop on 802.11g wireless. Maximum throughput on the Buffalo is nearly twice as fast as the Linksys:

17.5 mbit – Linksys WRT54GL running DD-WRT v24-sp1
32.5 mbit – Buffalo WZR-HP-G300NH running OpenWRT 10.03

Command Summary

For those that want to quickly install a setup similar to this, just run these commands at the terminal. This is really just for my own notes, in case I need to quickly reinstall.

opkg update
opkg install kmod-ath9k wpad-mini
opkg install kmod-fs-btrfs kmod-fs-ext2 kmod-fs-ext3 kmod-fs-ext4 kmod-fs-isofs kmod-fs-reiserfs kmod-fs-vfat kmod-fs-xfs
opkg install kmod-nls-cp1250 kmod-nls-cp1251 kmod-nls-cp437 kmod-nls-cp775 kmod-nls-cp850 kmod-nls-cp852 kmod-nls-cp866 kmod-nls-iso8859-1 kmod-nls-iso8859-13 kmod-nls-iso8859-15 kmod-nls-iso8859-2 kmod-nls-koi8r kmod-nls-utf8
opkg install kmod-usb2 kmod-usb-storage kmod-usb-storage-extras block-hotplug block-mount hotplug2
opkg install --force-overwrite avahi-daemon bwm conntrack-tools fdisk fping iftop ip iptables-utils kmod-ipv6 libnl lft lsof luci-app-ntpc luci-app-openvpn luci-app-upnp luci-ssl miniupnpd mtr net-tools-hostname ngrep nmap ntpclient openssh-client openvpn rsync screen snmpd sshfs tcpdump vim

Warning: the ash command line is limited to 512 characters, which is a little under 7 lines at 80 characters wide. Beware of this when pasting long package lists or commands. The commands above all fit within that restriction.

Conclusions

The Buffalo WZR-HP-G300NH running OpenWRT 10.03 is a serious improvement over the Linksys WRT54GL with DD-WRT v24-sp1. It is capable of higher wired and wireless throughput, can support more software and more TCP/UDP sessions, and is more configurable. One drawback is that everything in DD-WRT is configurable through the web interface, whereas some settings in OpenWRT may only be configured at the command line. For example, DD-WRT offers a GUI for QoS. This is a complicated subject that benefits from a GUI, and in this area I think DD-WRT wins out. However, this is a symptom of features being introduced as packages faster than the GUI can be extended to support them. I’d rather have a feature with a text file interface than no feature at all.

A Buffalo WZR-HP-G300NH can be had from Amazon UK for £69, while a Linksys WRT54GL is £48. If you’re considering buying a new router, buy the Buffalo. If you are thinking about upgrading, 70 quid gets you a very capable improvement on your current router.

Updates

2010-09-30: See my next post for upgrading to the latest OpenWRT release. If you’re installing a new Buffalo WZR-HP-G300NH, I recommend installing 10.03.1-rc3 directly.

2011-10-21: I’ve upgraded to each release in the 10.03.1-rc series since this was written. I advise you to directly install the latest release. See all posts related to OpenWRT.