tl;dr? Here’s how to fix it:

1. Login to the Nas4Free web interface.
2. Navigate to “Disks” -> “ZFS” -> “Volumes”.
3. Create a ZFS Volume called “swap”. For size, “2x RAM” is the standard. Otherwise use defaults.
4. Navigate to “System” -> “Advanced” -> “Swap”.
5. Set “Type” to “Device”, and “Device” to “/dev/zvol/tank/swap”, where “tank” is your ZFS pool name and “swap” is the name of the volume you just created as a swap device.

If you don’t already have ZFS configured, first create your ZFS pool as normal. Then follow the above steps.

Nas4Free allows you to install “embedded” or “full” software installations. I chose “embedded”, because this makes upgrades very easy, and it is simple to install this way on a USB flash device. This means you can dedicate all SATA ports to ZFS storage and not the operating system. With the embedded installation, Nas4Free does not create a swap space. Which means it’s up to the user to do this. It’s possible this isn’t an issue on the “full” installation. I don’t know; I haven’t tried it.

Why is swap necessary? After all, if you have a large amount of RAM, you shouldn’t need swap. And if you use ZFS, you should always have a large amount of RAM. That’s the ideal situation, but in practice it doesn’t work. Why? Because ZFS uses any available memory for cache.

Memory usage with ZFS on FreeBSD.

Under FreeBSD, this is called wired memory. When ZFS uses wired memory as cache, it should be reclaimed for other uses when it’s needed. That’s certainly what happens under Linux, where disk (read) cache is reported as “Cache memory” in the above graph.

I have found this isn’t the case in practice. Instead, when memory gets tight, FreeBSD kills processes it really shouldn’t:

Apr 30 12:49:45 san1 kernel: pid 69430 (istgt), uid 0, was killed: out of swap space

That’s right. Run ZFS for long enough with no swap, and eventually your iSCSI daemon will be killed. Which means your SAN isn’t accessible any longer. It could also kill the NFS server if you are using the userspace NFS daemon, which is what Nas4Free does.

FreeNAS, from which Nas4Free is forked a similarly-named commercial product, doesn’t have this problem. It defaults to using a certain amount of each disk for swap, which means the average user simply won’t see this problem. This may be a moot point, since the average user in this situation should be a highly-trained engineer. But it happened to me, and I’m documenting it here so it doesn’t happen to you.

Thanks to the fine people of the Nas4Free forums for providing additional information.

Pascale loves the Raspberry Pi logo, and calls the new computer “Raspberry Pi … Pi Pi Pi!” I installed Raspbian on the SD card the night before. The next morning, we sat down together with all the parts and cables. We watched the video and assembled the enclosure together. Then Pascale figured out all where all the cables go and how they snap in. We’re still waiting for the HDMI-to-DVI cable to connect it to the screen on her table, so we tested it on the TV.

We bought the parts on eBay. Here are the details:

1. Raspberry Pi model B 512MB RAM
2. Adafruit Pi Box, a clear acrylic enclosure
3. 16 GB SD card with the latest Raspbian

We already had the peripherals:

1. GMYLE Super Slim USB 2.0 Mini Keyboard
2. Sweex MI055 Mini Optical USB Mouse, Green
3. ViewSonic VX715 display with DVI and VGA inputs

Tomorrow, we’ll figure out Flash on the Raspberry Pi, so she can visit her favourite website.

This is really just a user interface bug. QuickSSHd is still starting dropbear, it just can’t read the server PID file to verify that it is running. The workaround is to rename the “dropbear” binary in /data/data/com.teslacoilsw.quicksshd/dropbear/, and replace it with a script which adjusts the umask before calling the real dropbear.

So start QuickSSHd once, and ignore the appearance that it’s not running. Then login as root (or use su in a terminal) and run:

cd /data/data/com.teslacoilsw.quicksshd/dropbear/
cp -a dropbear dropbear.real
cat > dropbear << END_OF_LINE
#!/system/bin/sh
umask 000
/data/data/com.teslacoilsw.quicksshd/dropbear/dropbear.real "$@"
END_OF_LINE

The cp -a ensures that the new dropbear script is owned by the right user and have the right permissions. If not, you will need to use chown and chmod to fix it. The right user/group will be different on each Android device, and the permissions should be the same as the original dropbear (now dropbear.real).

After applying this fix, you will need to reboot your phone, or kill the running dropbear using ps and kill. After that, QuickSSHd should work as normal.

Csync processing step propagate failed

Starting the client with “owncloud --logwindow” produced this message:

03-18 06:36:43:712 _csync_propagation_file_visitor: FAIL NEW: Documents/stuff.tgz
03-18 06:36:43:712 csync_propagate: Propagation for remote replica took 15.26 seconds visiting 497 files.
03-18 06:36:43:712  #### ERROR during  cysnc_reconcile :  "CSync processing step propagate failed.<br/>
Backend Message: Could not read response body: connection was closed by server"

After some digging, I discovered:

1. It only happens with Apache, and not with Nginx.
2. It happens with Csync 1.2.0 and up, but not Csync 1.1.x.
3. It happens after upgrading to ownCloud 4.5, and may not affect ownCloud 4.1.
4. It only happens if you add compressed files, including at least zip and tar.gz.

Possible solutions include downgrading Csync or switching web servers, but you don’t really want to do that. The easy workaround is to disable the “Archive support” plugin and force ownCloud to recalculate the file cache.

1. Disable Archive support
1. Login to the ownCloud web interface
2. Click “Settings” -> “Apps”
3. Select “Archive support”
4. Press “Disable”
2. Force ownCloud to regenerate the file cache
1. In ownCloud, click “Files”
2. In Firefox, go to the Firefox or Tools Menu
3. Select “Web Developer” -> “Web Console” to open a Javascript console
4. Type “scanFiles(true)” and hit Enter

After that, Csync should synchronise your compressed files correctly.

rm: cannot remove `/run/user/root/gvfs': Is a directory

This happens because the root user has started a gvfsd daemon, which mounts a virtual filesystem on that directory. The root user should never run GNOME, so it should never need gvfsd. But something in Ubuntu 12.10 starts one sometimes.

You can work around it by unmounting this directory. However, I prefer the more direct approach of killing root’s gvfsd.

First create a shell script:

sudo tee /usr/local/bin/kill-root-gvfsd > /dev/null << ENDOFLINE
#!/bin/sh
killall -u root gvfsd 2>/dev/null
exit 0
ENDOFLINE
sudo chmod 755 /usr/local/bin/kill-root-gvfsd

* Like my tee trick? Sudo is great, but I hate not being able to use shell redirects.

Running this script will kill root’s gvfsd daemon. You can run this script whenever the problem happens. But since the issue is random, and since I never want to see it again, I’ll use cron to run it periodically.

sudo tee /etc/cron.d/kill-root-gvfsd > /dev/null << ENDOFLINE
0,30 * * * * root [ -x /usr/local/bin/kill-root-gvfsd ] && /usr/local/bin/kill-root-gvfsd
ENDOFLINE

Now gvfsd will stay dead. Or at least never live longer than 30 minutes.

This is a hack. A good solution would be to find out why gvfsd starts and run this script afterward. A better solution would be to find a way to prevent root from starting gvfsd at all. As always, suggestions are welcome.

Here is the default graph:

(Click to embiggen)

The implementation for “used memory” in the SNMP OID is inconsistent across devices. But you can fetch “total memory”, and do the math:

Used memory = Total – (Free + Cache + Buffers)

Many people have used this method to make better memory usage templates. Eric A. Hall made a much prettier graph, but it relies on an external script to fetch the data and do the math. This is unnecessary, and (slightly) slower than using Cacti CDEF functions. Hans Fugal used CDEF functions, but his graphs use an eye-searing colour scheme. He also uses the wrong base; there are 1024 bytes in a kilobyte of RAM, not 1000 as in hard drives or network data rate.

Here are mine:

My implementation uses Eric Hall’s colour scheme and Hans Fugal’s CDEF method, but graphs real memory and swap as two separate graphs. It also uses the correct unit base of 1024. These graphs are also intended for Linux. Unix operating systems won’t report Buffers or Cache, but they’ll graph correctly as long as they report Free and Total.

Downloads:

cacti_graph_template__ucdnet_-_memory_usage_real.xml

cacti_graph_template__ucdnet_-_memory_usage_swap.xml

Only after making my own did I discover fmangeant‘s graphs. These use the CDEF method and correct base, although I’m not fond of the colours. The “Memory Usage Unix” graph would be useful if your device doesn’t report Buffers or Cache and you don’t want to see zero values on your graph.

Here is a disorienting upside-down shot of the Super Power Bank so you can read the technical details:

I knew I could answer Soren’s question by simply measuring the recharge time between known points (say, 10% to full) on first one port, then the other. But where’s the fun in that? I wanted to understand why and how some USB devices exceed the 500 mA limitation of USB 2.0 ports.

So, here is the USB test harness I built:

This is a 50x35x20 mm project box from Maplin. Through it runs a common 1m USB type A male to type A female cable, typically called a “USB extension lead”. In the project box I’ve installed six 2mm test sockets in various colours. I cut and soldered the wires to the test sockets, and maintained the earth connection between the outer plug shields. These are wired as so:

I’m interested in measuring the current and voltage of the power pins, so there are two sockets in series for each line. To measure current you must insert your multimeter in series. However, these series sockets interrupt the power flow, so to actually use the cable I must bridge the gap with a test lead or multimeter. I’m only interested in voltage of the data pins, so there is one socket for each line, in parallel. I’m also interested in shorting the data pins, as some USB devices apparently take that as a sign they can draw more than 500 mA (PC motherboard data pins would never be shorted). To do this, I can use the blue test lead to bridge between the yellow and blue sockets.

Here is the USB test harness in use, not charging a bluetooth keyboard (it has a full battery):

Testing with my multimeter, I learned that the Super Power Bank does not short the data pins. Instead, it outputs different DC voltages. Both ports output 0.64 V on the data pins when idle. Under load, the 1.0 A port outputs 1.92 V, and the 2.1 A port outputs 2.35 V. There doesn’t appear to be a standard for this, although Wikipedia has some ideas. The Super Power Bank doesn’t actually begin charging a device until you press its charge button. The power pins, as expected, show 0 V when not charging, and +5 VDC after you press the button.

When you connect a USB device to charge and press the button, the device seems to negotiate for a moment before drawing current. My HTC Desire HD draws about 0.2 A at first, then if it needs charging this jumps to 0.45 A. This is regardless of which port (1.0 or 2.1 A) I use. If the data pins are shorted, however, it instead draws 0.54-0.58 A.

With the help of Travis, I tested various Android devices charging behaviours. All devices had at least partially discharged batteries. All results show Amperes at 5 VDC, as measured by my multimeter’s inline 10A circuit. Tests were performed on each port of the Super Power Bank, with data pins either normal or shorted.

Update 2013-02-11: Gathered Nook Color and iPhone 5 data.

Conclusions:

• Some devices (Nexus 7, Nook Color) don’t exceed 500 mA even if the port indicates it can provide more.
• The Desire HD, Galaxy Nexus, and Nexus S all accept shorted data pins as permission to draw more than 500 mA.
• The Nexus 4 and iPhone 5 accept data pin voltage of at least 2.35 V as permission to draw more than 500 mA.

I gave the USB test harness to Soren today, at the moment this post is scheduled to appear on my blog. Surprise, Soren! Thanks for the cool toy and for the inspiration to do science!

The Solution

For the “tl;dr” crowd, here’s the solution. Add the following undocumented variable to /etc/default/dnsmasq:

DNSMASQ_EXCEPT=lo

And restart dnsmasq:

service dnsmasq restart

The Problem

Using your own dnsmasq server has been difficult since Ubuntu 12.04, because network manager runs a local dnsmasq resolver on localhost. The solution, to add bind-interfaces to dnsmasq.conf, is documented here. As of Ubuntu 12.10, this is now done by default in /etc/dnsmasq.d/network-manager.

However, Ubuntu 12.10 introduces a new problem. The /etc/init.dnsmasq script registers 127.0.0.1 as a resolver with resolvconf, which manages the DNS server list in /etc/resolv.conf. It does this regardless of the configured listening interfaces in the dnsmasq configuration. So even if you aren’t listening on localhost, it’ll tell resolvconf that it is. This wasn’t a problem before, since the dnsmasq instance started by network manager listens on 127.0.0.1. But as of Ubuntu 12.10, network manager’s dnsmasq listens on 127.0.1.1. So when then dnsmasq init script starts, it tells resolvconf that a DNS server exists on 127.0.0.1 even though it doesn’t. Then resolvconf chooses that as the best choice resolver, and poof – no DNS.

This is only a problem if you run an instance of dnsmasq which does not listen on localhost, as I describe in Using host networking and NAT with VirtualBox, v2.0.

The best solution is to add real support for updating resolvconf to dnsmasq. Barring that, someone should fix /etc/init.d/dnsmasq to be more careful about what it tells resolvconf. However, my solution works just fine. It forces /etc/init.d/dnsmasq not to inform resolvconf that it is available to answer DNS on localhost. This keeps DHCP and DNS for your virtual machines separate from DNS for the host operating system.

WARNING: This is dangerous. The Nmap Scripting Engine (NSE) allows scripts to sniff the network, change firewall roules and interface configuration, or exploit vulnerabilities including on localhost. It’s possible, especially with elevated capabilities, for a clever person to use Nmap and NSE to escalate to full root privileges. If you do not understand these risks, do not do this.

Nmap can perform much of its functionality as a normal user, so it checks if it has permissions to perform certain actions before falling back to safer behaviours. For instance, performing a TCP SYN scan (-sS) requires opening raw sockets. If this fails and if the user didn’t explicitly specify -sS, Nmap falls back to a normal TCP connect scan (-sT).

You can configure Nmap to use Linux capabilities just like Wireshark. However in certain circumstances (such as specifying a source port less than 1024), Nmap also needs CAP_NET_BIND_SERVICE.

sudo setcap cap_net_raw,cap_net_admin,cap_net_bind_service+eip /usr/bin/nmap

Some NSE scripts may require additional capabilities, but these three should suffice.

You must explicitly tell Nmap that it has these capabilities:

nmap --privileged -sS 192.168.0.1

You can save the extra typing by setting the NMAP_PRIVILEGED environmental variable.

export NMAP_PRIVILEGED=""
nmap -sS 192.168.0.1

There are many places to do this. Add the export line to one of the following:

1. ~/.bashrc, for bash
2. ~/.profile, for most shells as well as bash
3. ~/.xsessionrc, for most graphical environments
4. ~/.gnomerc, for GNOME only

I prefer .xsessionrc, since it will be inherited by any program I launch, including terminals and shells. However, some versions of LightDM, the login manager used by Ubuntu since 11.10, don’t source this file. An easy workaround for those users is to upgrade to Linux Mint use .gnomerc instead.

As with other programs using elevated capabilities, you should restrict Nmap to a certain group:

chgrp adm /usr/bin/nmap
chmod 750 /usr/bin/nmap

Unfortunately Zenmap, Nmap’s GUI, ignores the NMAP_PRIVILEGED environment variable. On startup, it will complain that “You are trying to run Zenmap with a non-root user! Some Nmap options need root privileges to work.”

If you used the .xsessionrc or .gnomerc methods above, Zenmap and the Nmap instances it calls will inherit NMAP_PRIVILEGED variable. So you can dismiss the annoying warning and use Zenmap as if you are the root user. I submitted a patch to nmap-dev, which was accepted and will be available in a release after Nmap 6.25. If your release doesn’t have this fixed yet, apply it with:

wget -q http://www.tolaris.com/blog/wp-content/uploads/2013/01/NMAP_PRIVILEGED.patch -O- \
| sudo patch /usr/share/pyshared/zenmapGUI/App.py

You can now run Nmap and Zenmap as an unprivileged user without warnings.

Update 2013-01-29: At Fyodor’s request, I’ve rewritten this guide for Secwiki.org.

root@laptop:~# tshark 
tshark: Lua: Error during loading:
 [string "/usr/share/wireshark/init.lua"]:45: dofile has been disabled
Running as user "root" and group "root". This could be dangerous.

Although I’ve long been aware of the dangers of running Wireshark as root, the convenience has outweighed the danger until now. The annoyance of this new output finally prompted me to act. The solution is to use Linux capabilities to allow Wireshark to sniff without using any of root’s other permissions.

There are plenty of instructions out there telling people how to do this. Ubuntu users (since 10.04) have a very easy method. All solutions involve granting the CAP_NET_RAW and CAP_NET_ADMIN capabilities to any binary you want to allow to sniff:

sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

However, this allows all local users to sniff. On a normal workstation, that’s probably fine. But if you want to be more secure, you can restrict access to /usr/bin/dumpcap to certain groups. “adm” is a good choice for most distributions, as console users are generally members. Alternately, consider the group “sudo” (Ubuntu 12.04 or later), “admin” (Ubuntu before 12.04), or “wheel” (Red Hat systems).

sudo chgrp adm /usr/bin/dumpcap
sudo chmod 750 /usr/bin/dumpcap

In addition to wireshark, there are a number of other programs that I like to run as an unprivileged user. For my notes and yours:

for i in `which dumpcap iftop ngrep tcpdump tcptraceroute traceroute.db` ; do
sudo setcap cap_net_raw,cap_net_admin+eip $i
done

nmap is a special case. More on that later now.

The only problem I’ve found with this method is that unprivileged users still cannot sniff USB packets. For now only root can do that.