• Iron Blogger Freiburg

    A blog entry from Antischokke made me aware of a great idea to breathe new life into the blogs of local writers. It’s called Iron Blogger, a group effort that requires the participants to write at least one blog post per week. Otherwise, they’ll have to chip in a “fine”. Every so often, the fines will be converted to drinks collectively.

    I like this idea and invite all bloggers in Freiburg to join Iron Blogger Freiburg! We’ll use Mako’s rules. The fine will be 4€ per missed post (payable in person or via PayPal), and I’ll organize a meetup when the beer pool reaches 40€. The slacker limit is 20€ (reach it and you’re out unless you pay the balance).

    If you’d like to join the group, let me know. You’ll reach me via the comments below, via email or Twitter.

    I’m looking forward to get this thing off the ground. There’s some writing to do and I just ordered myself a new keyboard!

    UPDATE: Woohoo! It's taken only a few hours to get an enthusiastic group together! I've created a separate page on this website for us.

  • Make temporary files non-executable

    At DrupalCONCEPT operations, our intrusion detection system recently notified us that it found a rootkit in the directory /dev/shm on one of our servers. This directory is writeable by the Apache webserver, so attackers that find a vulnerability in the installed software are able put hostile content (aka rootkits) there.

    Of course, the vulnerability shouldn’t be there in the first place. We’re doing security updates all the time, but only on the OS and hosting infrastructure level. Since the actual web applications running on our infrastructure (in our case, Drupal) are maintained by our customers, we don’t have the same kind of tight control here as we have on the OS level.

    Okay, we may not be able to prevent attackers from deploying their scripts. But we can prevent those scripts from doing any harm. This is where the noexec filesystem option comes in handy. Files on filesystems that have this option enabled can’t be executed even if they have their execution permissions (“x” ) set.

    We use a Chef recipe to modify /etc/fstab accordingly. The first execute resource does a remount of the /dev/shm filesystem, but only if triggered by another resource. Namely, the following bash resource that modifies /etc/fstab if it’s not already hardened:

    [gist id=1550976]

    Since we include this recipe in our base Chef role, it’s applied to every server we set up.

  • A great tutorial for creating Chef cookbooks

    System administrators who are looking for a tool that helps them automating their maintenance tasks and have no or only little experience with Chef should really take a look at Joshua Timberman's great tutorial "Guide to Writing Chef Cookbooks".

    In his article, Joshua describes all steps he takes to create a new Chef cookbook that installs and maintains smartmontools (a set of tools to monitor hard disk health). It's a great example how straightforward it is to automate systems operations tasks with Chef.

    Even with two years experience in using Chef, I learned one or two bits from this tutorial. And it just so happened this week that I needed a smartmontools cookbook. So, thanks twice for writing this up, Joshua!

  • How not to distribute DNS servers

    For one of our customers that addresses the south american market, we've rented a server at HostDime in Brazil. Unfortunately, they often suffer network outages.

    Once again, we can't reach our server, so I try to access their Ticket system named "Core". It's unreachable, too. Let's see:

    $ host core.hostdime.com.br
    Host core.hostdime.com.br not found: 3(NXDOMAIN)

    Okay, looks like DNS is down. But there's more than one DNS server, isn't it?

    $ host -t ns hostdime.com.br
    hostdime.com.br name server ns1.hostdime.com.br.
    hostdime.com.br name server ns2.hostdime.com.br.

    There is. So how...

    $ host ns1.hostdime.com.br
    ns1.hostdime.com.br has address 187.45.182.3
    $ host ns2.hostdime.com.br
    ns2.hostdime.com.br has address 187.45.182.4

    m( Does anyone have a suggestion for a hosting provider in Brazil that's not run by idiots?

  • Travel tips for sysadmins

    OpenDNS recently added a datacenter location in Frankfurt, Germany. On their blog, George Patterson, Director of Operations for OpenDNS, not only posted some pictures of their server rack but also a bunch of tips for sysadmins that have to travel to a remote facility:

    • Have a solid deployment checklist of everything you want at the site. If you don't bring all necessary tools and equipment with you, getting them will cost you extra time.
    • Set up all your power at the datacenter and make sure it's working before you leave. Don't waste time waiting for the datacenter staff to have your power supply connected. And have them install a remote manageable power distribution unit, so you don't have to pay remote-hands charges.
    • If you can avoid it, don’t book a flight until your gear has cleared customs. Depending on the country, customs handling can take from a few days to several weeks. Don't just hope that your gear will arrive earlier than you.
    • Always plan for extra days. You shouldn't have to go into fast-forward mode because something took a bit longer than planned; that will only account for more problems. Plan for some extra days and if you'll finish early, there probably will be more to go and see than only a datacenter.
    • Take photos along the way, and at the end. If your site documentation includes images, it's very easy to point a remote tech to the right place.

    Read George's whole blog post on the OpenDNS blog!