Devops podcasts

While there hasn’t been a shortage on IT engineering blogs for a long time, podcasts that deal with devops topics are a rare sight. That’s why I’d like to recommend the ones that I currently subscribe to:

  • DevOps Cafe Podcast: Damon Edwards, John Willis and guests talk about interesting news in the datacenter world.
  • The Changelog: A show that “covers what’s fresh and new in Open Source”, hosted by Wynn Netherland and Adam Stacoviak
  • The Food Fight Show is a bi-weekly podcast for the Chef community, or, as hosts Bryan Berry and Matt Ray put it, “The Podcast where DevOps chefs do battle”
  • itkanban’s podcast comes also bi-weekly and covers news about lean and agile IT management methods.

Do you know any other podcasts a self-respecting system administrator should listen to? Please post them in the comments!

How to use a remote shell over flaky connections without losing your nerves

Recently, I’ve stumbled upon the Mosh remote shell application, and since then I can’t stop talking about it! If you need to access your servers’ command line interface over slow, unreliable connections, you want to use the Mobile Shell.

As the website describes it, Mosh is a…

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

And – spoiler alert – it works so well that I’ve completely replaced ssh for mosh for accessing our many servers. SSH is still necessary, though, because the Mosh client first opens an SSH connection to the target server and then launches its server component there. After establishing an UDP connection between the client and server components, the SSH connection is dropped.

By using a new protocol called the State Synchronization Protocol (SSP) which is based on UDP, Mosh provides a shell connection that’s far more usable over slow and flaky connections than SSH, for example when using a 3G network from a train. It even survives reconnects that change the client’s IP address. I was really amazed when, on my first day with Mosh, all shell sessions I started in Starbucks simply resumed after I opened my laptop again in my homeoffice.

Local Echo is another great feature that makes working over unreliable connections far less annoying. While SSH doesn’t display your keystrokes until they have been sent back from the server, Mosh shows them immediately without requiring the roundtrip. That way, you can spot and correct typos without wait and finally hit the Enter key with all confidence. It may take some time until you see the effect of your command due to your slow connection, but at least typing it was no hassle at all. Whily typing, Mosh gives you visual feedback about the synchronization process by underlining those parts of the command line that have not yet been acknowledged by the server.

Of course, Mosh also uses encryption, so you don’t lose any security by switching from SSH to Mosh. Installation is easy, too, so don’t wait any more. Start moshing!

Make temporary files non-executable

At DrupalCONCEPT operations, our intrusion detection system recently notified us that it found a rootkit in the directory /dev/shm on one of our servers. This directory is writeable by the Apache webserver, so attackers that find a vulnerability in the installed software are able put hostile content (aka rootkits) there.

Of course, the vulnerability shouldn’t be there in the first place. We’re doing security updates all the time, but only on the OS and hosting infrastructure level. Since the actual web applications running on our infrastructure (in our case, Drupal) are maintained by our customers, we don’t have the same kind of tight control here as we have on the OS level.

Okay, we may not be able to prevent attackers from deploying their scripts. But we can prevent those scripts from doing any harm. This is where the noexec filesystem option comes in handy. Files on filesystems that have this option enabled can’t be executed even if they have their execution permissions (“x” ) set.

We use a Chef recipe to modify /etc/fstab accordingly. The first execute resource does a remount of the /dev/shm filesystem, but only if triggered by another resource. Namely, the following bash resource that modifies /etc/fstab if it’s not already hardened:

Since we include this recipe in our base Chef role, it’s applied to every server we set up.

A great tutorial for creating Chef cookbooks

System administrators who are looking for a tool that helps them automating their maintenance tasks and have no or only little experience with Chef should really take a look at Joshua Timberman’s great tutorial “Guide to Writing Chef Cookbooks“.

In his article, Joshua describes all steps he takes to create a new Chef cookbook that installs and maintains smartmontools (a set of tools to monitor hard disk health). It’s a great example how straightforward it is to automate systems operations tasks with Chef.

Even with two years experience in using Chef, I learned one or two bits from this tutorial. And it just so happened this week that I needed a smartmontools cookbook. So, thanks twice for writing this up, Joshua!

How not to distribute DNS servers

For one of our customers that addresses the south american market, we’ve rented a server at HostDime in Brazil. Unfortunately, they often suffer network outages.

Once again, we can’t reach our server, so I try to access their Ticket system named “Core”. It’s unreachable, too. Let’s see:

$ host core.hostdime.com.br
Host core.hostdime.com.br not found: 3(NXDOMAIN)

Okay, looks like DNS is down. But there’s more than one DNS server, isn’t it?

$ host -t ns hostdime.com.br
hostdime.com.br name server ns1.hostdime.com.br.
hostdime.com.br name server ns2.hostdime.com.br.

There is. So how…

$ host ns1.hostdime.com.br
ns1.hostdime.com.br has address 187.45.182.3
$ host ns2.hostdime.com.br
ns2.hostdime.com.br has address 187.45.182.4

m( Does anyone have a suggestion for a hosting provider in Brazil that’s not run by idiots?