When you login via SSH with agent forwarding enabled, the SSH server creates a socket in /tmp/ssh-[HASH]/agent.[PID]. This allows SSH sessions on the remote host to access your agent back on your workstation. Commands that use the agent, like ssh, scp, and rsync, find it by reading the SSH_AUTH_SOCK environment variable.

root@server:~#  echo $SSH_AUTH_SOCK
/tmp/ssh-NshdZD1538/agent.1538

This path is readable only to the current user (and root, which is why agent forwarding is dangerous if you don’t trust the root users of servers you access). GNU Screen inherits SSH_AUTH_SOCK like any other process. However, what happens when you detach your screen session and logout? When you login again and re-attach to screen, SSH_AUTH_SOCK is pointing to the wrong file:

root@server:~# screen -r
root@server:~# echo $SSH_AUTH_SOCK
/tmp/ssh-NshdZD1538/agent.1538
root@server:~# ls /tmp/ssh-NshdZD1538/agent.1538
ls: cannot access /tmp/ssh-NshdZD1538/agent.1538: No such file or directory
root@server:~# ssh localhost
root@localhost's password:

I have found a number of solutions of varying complexity. But here is a simple one-liner which finds an agent socket and attaches to it. This works in the current screen session (not just new shells within that session), and you don’t have to modify .bashrc or switch to zsh to use it.

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* | tail -n 1)

This finds the most recent SSH agent owned by the user, and assigns it to SSH_AUTH_SOCK. This doesn’t guarantee that you’ll attach to the socket created for your current SSH session, only the most recent one. For instance, you could login with three SSH sessions and this will find the latest one. But if you’re using screen, you only really need one SSH session anyway.

Here is attempt number 3, now with more complicated regular expressions:

#!/usr/bin/perl
# parse URL
($protocol,$host) = split /:\/\//, $ARGV[0];
($host,$port) = split /:/, $host;

# validate input
if ( $protocol !~ /^(telnet|ssh)$/ ||
   $host !~ /^[a-zA-Z0-9][a-zA-Z0-9.-]*$/ ||
   $port !~ /(^[a-zA-Z0-9][a-zA-Z0-9_-]*$|^$)/ ) {
        warn "Invalid URL";
        exit 1;
}

# if SSH, add -p argument
if ( $protocol eq "ssh" && $port != '' ) { $port = "-p $port" ; }

# call terminal emulator
exec("konsole --hold -e $protocol $host $port");
exit;

Your move, sir.

You can download an updated url-terminal script here. You can read the post that started this here.

Update 2010-08-27: don’t use this script either! See my next post for a better one.

#!/usr/bin/perl
# parse URL
($protocol,$host) = split /:\/\//, $ARGV[0];
($host,$port) = split /:/, $host;

# validate input
if ( $protocol !~ /^(telnet|ssh)$/ || $host !~ /^[a-zA-Z0-9.-]+$/ || $port !~ /(^[a-zA-Z0-9_-]+$|^$)/ ) {
        warn "Invalid URL";
        exit 1;
}

# if SSH, add -p argument
if ( $protocol eq "ssh" && $port != '' ) { $port = "-p $port" ; }

# call terminal emulator
exec("konsole --hold -e $protocol $host $port");
exit;

This script only accepts telnet:// and ssh:// URLs, where the host is a valid domain name and the port is a valid port (including text aliases like “smtp” from /etc/services). It passes SSH port arguments correctly and tells Konsole to stay open after the session terminates.

You can download a more advanced form of this script here.

As with most tinkering in firefox, start by typing “about:config” in the location bar. Right click and select “New”, then “Boolean”. Create two entries:

network.protocol-handler.expose.telnet = false
network.protocol-handler.expose.ssh = false

Now, click on a telnet or SSH URL, and Firefox will prompt you for the application to use. This application must handle the full URL as an argument. On Linux, the easiest solution is to choose /usr/bin/xdg-open. This will open the user’s preferred terminal, whether that is gnome-terminal, konsole, or xterm. You can use xdg-open to open almost any type of file or URL.

Alternatively, choose /usr/bin/putty, or use a simple script as follows. Edit the last line to call whatever application you prefer.

Update 2010-08-25: don’t use this script. See my next post for a better one.

#!/usr/bin/perl
# take URL of form telnet://target:port and call konsole

# get protocol and host
($proto,$addr) = split /:\/\//, $ARGV[0];

# convert "host:port" to "host port" (port is optional)
$addr =~ s/\:/\ /g;

`konsole -e $proto $addr\n`;

A bit of history, for the curious. You may find instructions online stating to create values like these:

network.protocol-handler.app.telnet = "/usr/bin/putty"
network.protocol-handler.warn-external.telnet = false

This is the old method, used in releases prior to Firefox 3.5. These settings are now ignored.

1. save passwords
2. manage connections via the GUI
3. support VNC and RDP
4. work with our IP KVM

This last requirement is the kicker. When I change hosts on our Adderview IP KVM, it draws one frame at 0×0 resolution, and then changes to the resolution of the new host. This has crashed every VNC client I’ve tried except xvnc4viewer. Which of course, doesn’t save passwords or have a GUI (the raw X menu when you press F8 does not count).

I have tried KRDC, Vinagre, xtightvncviewer, and a number of simpler command-line VNC clients. They all seem to be designed to support the “my Mom needs tech support” problem, not the “I manage remote servers” problem. KRDC won’t even login to my KVM; it just hangs after authentication. I suppose that’s better than crashing, but it is still a show stopping bug for me.

Over a year ago, I hacked up a scripted solution involving zenity GTK dialogs, saved password files, and xvnc4viewer, but I’m not going to release that. Instead, meet the last remote desktop client you’ll ever need: Remmina.

Remmina meets all of my requirements, has tons of useful features, and works flawlessly. Not only that, it is extremely responsive. Key presses are limited only by the round-trip time, with no added delay from VNC itself. There are no redrawing or tearing problems. It supports full-screen mode and scrolling inside a view port. It can forward over SSH and will use SSH keys files or your SSH key agent. It can synchronise the clipboard between host and remote server; this is the first time I’ve actually seen that feature work reliably. It can even open a standard SSH or SFTP session, or host a VNC server.

Installation is easy. An older release is available in the Ubuntu Lucid Universe repository, or get the latest version from their PPA.

My opinion is that Lucid improves on Karmic in almost every way. That’s not saying much, since the KDE 4 upgrade has been so painful. We’re finally back to KDE 3.5 functionality, and speed is improving with each release. On the other hand, there are a lot of new features in the OS since Hardy – grub2, ext4, ecryptfs, upstart, kernel mode setting, compositing window management, Strigi indexing and Nepomuk semantic desktop. Some of these have dramatically improved performance, while others have increased system requirements just to add eye candy. It’s hard to evaluate KDE on its own, so I’m not going to focus on that.

I backed up my entire drive to an external USB drive formatted for ext3 using rsync. I then booted from the Kubuntu 10.04 AMD64 Desktop CD, and followed the default install options until the disk partitioning step. I always install with separate /, /home, and swap partitions (this normally makes upgrades easier, unless you are reformatting as I did here). I used ext4 for / and /home, and chose to encrypt my home directory. I then followed the rest of the steps and rebooted at the end.

Following the reboot, I used rsync to restore my lost files and most of my dotfiles – .mozilla, .gnome, .gconf, .Virtualbox, and the like. However, I did not restore .kde. Instead I manually copied only some configs, for KGPG, Akregator, and Kopete. The rest of my KDE apps I reconfigured from scratch. I did this because we use Kolab at work, which integrates with Kontact but can be fussy with local contacts files. As of Lucid, Kontact uses Akonadi to manage contacts. I expected trouble, and found it. More on that later.

Finally, since I have an encrypted home directory, I also encrypted swap and created a tmpfs on /tmp. I followed the steps in my guide, and rebooted with no problems.

What’s better:

The ext4 filesystem is noticeably faster. This is the reason I reformatted instead of upgrading. I used ext3 under Karmic, and really wanted to see if what I’d been hearing about the speedy new filesystem is true. It is. It’s faster for booting and it’s faster for reading large data. My Virtualbox virtual machines load in almost half the time, even compared to the same Virtualbox 3.2.6 release on Karmic. I suppose there could be other differences contributing to this but it really stands out. I haven’t run an fsck yet but others report that it is much faster as well.

ext4 has definitely improved performance of copying data inside my encrypted home directory. I barely notice the performance hit from using ecryptfs now. The only time I do is when using rsync to compare large directories (like when my backup process examines my mail archive).

There is a new touchpad control in KDE Control Center. This enables gestures, including two-finger scrolling (but not pinch-to-zoom, which I hope is forthcoming), and different actions for tapping in corners, multi-finger tapping, and so on. Still missing is tap suppression (accidentally tapping while typing), so I still use syndaemon. Create ~/.kde/Autostart/syndaemon.sh, make it executable, and insert:

#!/bin/sh
# Disable touchpad while typing to prevent accidental tapping.
/usr/bin/syndaemon -d -t -i 1

The device notifier plasmoid now has multiple actions when opening attached storage devices, and can be configured to automatically mount drives. This is a vast improvement.

The system tray plasmoid now obeys my auto-hide preferences. Under Jaunty and Karmic, some applications had overriding preferences that caused them to always be hidden or visible. For instance, it was impossible to make KGPG always visible. I frequently use KGPG, so this caused me to almost always have the system tray expanded to show all applications.

Firefox/KDE integration works very well. Open/save file dialogs use KDE, and menus and icons use KDE defaults. The printer dialog is still the native Firefox one.

virt-manager is vastly improved. The GUI is more responsive when connecting, is prettier, and has graphs for CPU, disk, and network I/O.

ClusterSSH works with KWin again. Since 8.10, ClustterSSH has been nearly broken in KDE. First, simply starting it caused copy (to clipboard or selection) to stop working in most QT/KDE apps. Second, the ClusterSSH master window would grab focus and prevent you from giving focus to any of its children xterms. This made it very hard to run commands on just one host without running it on all. I gave up and used various other techniques for managing my servers. But nothing beats ClusterSSH for managing 2-20 servers at once, and I’ve sorely missed it. Welcome back, old friend!

What needed tweaking:

The Oxygen window decoration theme still doesn’t colourise the active window. Open System Settings, go to “Appearance”, then the “Windows” side bar. Under the “Window Decoration” tab, choose “Oxygen”. Under the “Decoration Options” area, choose the “Fine Tuning” sub-tab. Check “Outline active window title”.

Hotkeys in Kmenu are ignored. I use a few quick-launch shortcuts, such as “Win+T” to start a terminal. You can set these when editing the K menu, but they are ignored by default. Open System Settings, select “Input Actions”, and then check “KMenuEdit”.

VirtualBox and virt-manager don’t play well together. I don’t use Xen or KVM on my desktop, but I do manage several KVM-based virtual machine servers. Thanks to the new “install recommends” preference in the package manager, simply installing virt-manager also installs libvirt-bin. This loads the kvm-intel or kvm-amd modules on boot, which then prevents VirtualBox from starting virtual machines, with the error “VirtualBox can’t operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot (VERR_VMX_IN_VMX_ROOT_MODE).”

I suppose this is really a problem with the “install recommends” behaviour. I’ve complained about that elsewhere, but I always repeat a good gripe when the opportunity presents.

The solution is to edit /etc/default/libvirt-bin and disable libvirtd:

start_libvirtd="no"

And for good measure, blacklist the modules. Create /etc/modprobe.d/local.conf and insert:

blacklist kvm-intel
blacklist kvm-amd

What still needs work:

Akonadi doesn’t start before Kontact tries to access it. This solution (autostarting “akonadictl start” at login) worked for me, although I (painfully) developed it independently. If only I had used Google.

openvpn with knetworkmanager still doesn’t work. I still prefer Gnome’s network manager applet, which works just fine with Kubuntu. Kill knetworkmanager, and start nm-applet. Next time you login, KDE will tell you that another network manager is running, and ask you if you still want to use Knetworkmanager. Say no. Also, OpenVPN support is more reliable under Lucid. Using Gnome network manager with Kubuntu Karmic, the OpenVPN service would periodically fail to start. Editing VPN preferences and then hitting OK sometimes resolved it, but at other times it was an annoying and random dance to make it work. This seems to be resolved under Lucid.

OpenOffice/KDE integration is improved since Karmic, but still has drawing bugs. In particular, the zoom slider in the lower right often disappears. It’s still there, and clicking in the area makes it reappear and zooms. I prefer the “100% / 75% / …” pull-down of the stock OpenOffice theme, however. This is a vast improvement over the Karmic integration, where simply dragging a spreadsheet tab in Calc crashed OpenOffice, but I’d like to see more development here.

Otherwise, my comments regarding Karmic still hold. Google Earth and Kwin play nicely, qtcurve (KDE/GTK integration) is awesome and no longer has the font bug, and Plasma and Kwin are faster and more stable. Lucid is no great leap forward and Kubuntu is still not an innovator among KDE distributions like Ubuntu is to Gnome. But it is an incremental improvement worth using if you prefer KDE.

I have had a Lucid repository since upgrading my media PC and servers. It now includes dfreer’s znes32 for AMD64 (still working on Lucid) and kregexpeditor (you can have it when you pry it from my cold, dead hands).

If you aren’t already using SSH keys and SSH agent, you should. SSH agent forwarding allows for secure sideways authentication. For example:

tyler@baal:~$ ssh -A root@server1
Last login: Fri Sep  4 15:47:02 2009 from network-192-168-1-2.example.com
Linux server1 2.6.24-22-generic #1 SMP Mon Nov 24 18:32:42 UTC 2008 i686

root@server1.bed.talia.net:~# scp /tmp/file root@server2:/tmp/
file                                           100%    0     0.0KB/s   00:00
root@server1.bed.talia.net:~#

Notice how I was never prompted for a password or SSH passphrase? That’s because in step one, I had an SSH agent already holding my passphrase. And in step two, scp on server1 securely forwarded the key request back to my workstation and server2 accepted it. No passwords. Fast and easy.

This allows me to improve security in several ways.

1. I don’t have to remember my root passwords. I am free to store them in an encrypted file or encrypted password store.
2. Because I don’t have to remember them, I don’t have to make them memorable. I generate all passwords with pwgen -B.
3. Because I don’t have to remember them, they don’t have to be the same. If I have two servers or 50, they can all have unique passwords.

Note for Ubuntu users: if you use a GPG key agent such as KGPG, you’ve probably already got an SSH key agent running. You just need to configure it to load your keys at login time.

Now, suppose you have a key that an automated script is using. Or you have a user that you want to be able to access your servers from work, but not from home? This is where key access limits come in. In ~/.ssh/authorized_keys:

from="*.example.com" ssh-rsa AAAAB3Pca[...]Fy== user@server.example.com

You can also limit by IP or have a list of limits.

from="*.example.com,192.168.*" ssh-rsa AAAAB3Pca[...]Fy== user@server.example.com

Now this key will only be accepted from servers whose IP addresses resolve to something in “example.com”, or whose IP address begins with “192.168.”. Note that using a hostname here means the user can’t login if DNS doesn’t work, or if reverse DNS records are wrong for your IP. For all but senior administrators, this is probably an acceptable risk.

Update 2009-09-07 13:49 UTC: As I’ve just discovered, you may have to list IPv4 addresses in their IPv6 form. For example:

from="::ffff:192.168.*,192.168.*" ssh-rsa AAAAB3Pca[...]Fy== user@server.example.com

diff root@server1:/etc/randomrcfile root@server2:/etc/randomrcfile

There are ways to do this with Kompare (a KDE diff app) and the fish:// Kioslave, but not on the command line. SSHFS makes it easy:

mkdir /tmp/server1 /tmp/server2
sshfs root@server1:/ /tmp/server1/
sshfs root@server2:/ /tmp/server2/
diff /tmp/server1/etc/randomrcfile /tmp/server2/etc/randomrcfile

I do this so often that I’ve written my own script to handle this for me, smount. Copy it somewhere in your path, and optionally make a copy or symlink called “sumount”. Now you can rapidly mount and unmount multiple hosts.

List hosts:
smount

Mount two hosts:
smount server1 server2

Unmount all hosts:
smount -a

Help:
smount -h

Caution: this script is a dirty hack written in bash script. It assumes that it is free to create /tmp/hostname, and it doesn’t play well on a multi-user workstations (I don’t want to mount in /tmp/tyler@host or something like that). It also assumes that you have root access on your remote machines, so you probably want to use SSH keys.

smount works on Ubuntu hardy. However, I’ve found that the output of “mount” listings for FUSE filesystems changes from release to release. So don’t be surprised if “smount -a” stops working when you upgrade Ubuntu. I’ll try to keep an up-to-date copy here.

If you don’t know what you’re doing with mount or SSHFS, or if you’ve never totally destroyed a filesystem by accident (and are thus extremely cautious about doing it a second time), then DO NOT use this script. No warranty or fitness for purpose is implied. Your mileage may vary. RTFM.