Paul Goscicki

Some minor problems:

• /etc/default/locale has been deleted (wtf?). Needed to be recreated.
• Both /etc/timezone and /etc/localtime have been deleted. Needed to recreate the links.
• /etc/updatedb.conf has been deleted. Needed to be copied from another machine.

and one major one:

• klogd now takes 5 minutes to start, which means I have to wait 5 minutes after each reboot to use the machine. Adding -x switch in the init.d script solved the problem. What was the root cause? No idea. There are only hints.

Apparently there is some reasoning behind not upgrading your linux policy.

Does the following error sound familiar?

** !!! PID file log/mongrel.pid already exists.  Mongrel could be running already.  Check your log/mongrel.log for errors.
** !!! Exiting with error.  You must stop mongrel and clear the .pid before I'll attempt a start.

It usually happens when the server crashes. After that you need to ssh into it, remove the mongrel pid files and start the cluster manually. No more.

I assume you have mongrel_cluster setup properly, ie: project’s config file is in /etc/mongrel_cluster and the mongrel_cluster script has been copied from:
/usr/lib/ruby/gems/1.8/gems/mongrel_cluster-*/resources/
to the /etc/init.d directory. You need to edit the /etc/init.d/mongrel_cluster file:

Change this two bits:

start)
  # Create pid directory
  mkdir -p $PID_DIR
  chown $USER:$USER $PID_DIR

  mongrel_cluster_ctl start -c $CONF_DIR
  RETVAL=$?
;;

and

restart)
  mongrel_cluster_ctl restart -c $CONF_DIR
  RETVAL=$?
;;

to

start)
  # Create pid directory
  mkdir -p $PID_DIR
  chown $USER:$USER $PID_DIR

  mongrel_cluster_ctl start --clean -c $CONF_DIR
  RETVAL=$?
;;

and

restart)
  mongrel_cluster_ctl restart --clean -c $CONF_DIR
  RETVAL=$?
;;

respectively.

Adding the --clean option makes the mongrel_cluster_ctl script first check whether mongrel_rails processes are running and if not, checks for existing pid files and deletes them before proceeding.

You must be using the mongrel_cluster version 1.0.5+ for it to work as advertised (previous versions were buggy). To upgrade do:

gem install mongrel_cluster
gem cleanup mongrel_cluster

Here’s the related mongrel_cluster changeset.

During the recent move of my development machine to the basement I’ve conducted a test telling me what is the actual power consumption of my new Core 2 Duo powered server. Basically, it is a normal PC: Core 2 Duo E6300 1.86GHz, 2 x 512MB DDR2, 2 x 250GB SATA 7200rpm (RAID 1), old PCI graphic card and a 350W power supply. All running latest Ubuntu (currently 7.04 Feisty Fawn, server edition). Since it is a development machine, it’s idle most of the time (98% or even more). And this is the state I was making my measurements in. So what are the results? Well, I was quite surprised how low my power consumption actually is. I took three tests, which indicated basically the same: about 77 Watts. Even taking into the account temporary power usage spikes (when I’m actually using the machine…) it shouldn’t cost me more than 4$ per month to keep it running 24/7. Isn’t that sweet?

Actually it was more like I was loosing my mind, not my Ubuntu…

But let’s start from the beginning… I have two identical 250GB hard disks so I’ve decided to create a RAID array out of them. Not a system (bootable) one as I had too much trouble setting it up (I’ve set it up but dist-upgrade broke it all too nicely; kernel panic, etc.). I’ve set up a separate 5GB system partition on the first drive, leaving the rest for RAID. This left me with 5GB of free space to spare on the second drive. Smart as I was, I decided to clone the system partition from first drive to the second one, using dd, so I’d still be able to boot if either of the drives crashed. I called it semi-RAID built-by-hand and, well, I was quite proud of it. All seemed fine as months passed (and remember, that this was a server and as such almost did not require any reboots). But time passed and suddenly there was the new Ubuntu out, the Feisty one, so I decided it was time to upgrade. As I had some minor troubles during the upgrade (obsolete packages, invalid config files that I ordered to keep, etc.), I was rebooting every few minutes. And this is where fun comes in…

After a successful upgrade to 7.04 the screen greeted me with a 6.04 prompt. Hmm… strange. Let’s see what’s going on. Okey, so this upgrade actually did go so well. No problem, let’s do it again. This time I did not reboot, but kept making other changes. At some point I had to reboot, though. Now I was scratching my head really hard. Some packages, I knew I had uninstalled previously, kept coming back. I was making changes to various config files only to see those changes not being written to the disk after rebooting. Like, WTF? Now I was rebooting like crazy… loosing my mind more with every reboot. I was making directories like THIS-IS-FIRST-HARD-DISK-FOR-SURE only to see them disappear and reappear a couple of reboots later. I was almost crying with despair. I’ve came up with the idea to compare /dev/sda1 with /dev/sdb1. Funny thing, they turned out to be the same. Who knew, maybe my RAID-by-hand automatically turned into a real one?

I had dark thoughts. I was thinking about giving up on having two identical hard disks inside one PC and maybe about downgrading to Edgy, not even knowing whether that was possible. I was even thinking about giving up on those two 250GB disks. I was really desperate. I knew, I needed a break.

10 minutes and one glass of cold water later I was on the mission to find out what is exactly wrong with my Ubuntu. Or my PC. Or my hard disks. Or the world around me.

It wasn’t easy. The df command reported my system being on /dev/sda1. Mounting /dev/sdb1 did not help as it has been showing me the same partiton. But then came the bright idea to try and mount /dev/sda1, despite it being already mounted. To my surprise it turned out to be a completely another partition! The lost one! The one I missed so much. I was in heaven, so I started googling, because by that time I just knew it had something to do with those weird UUIDs. And I’d found out that I was not alone. I was so happy…

Now I know that my mistake was to make the exact clone of the system partition and have those two partitions (with the same UUIDs, yeah, unique ids my ass) available at the same time. No wonder my Ubuntu felt schizophrenic, but it still does not justify all of this weird behavior I was greeted with. Some error, some syslog entry, anything would be helpful… is that too much to ask?

What I was left with after I’ve figured it all out was this nice free disk space report (notice the double /dev/sda1 entry):

$ df
Filesystem  1K-blocks     Used  Available  Use% Monuted on
/dev/sda1     5162796  1650512    3250028   34% /
(..)
/dev/sda1     5162796  1558632    3341908   33% /mnt/disk-a

The root of the problem is that I base most of my core linux knowledge on the RedHat from the 90s when /dev/hda1 was saint and meant exactly what it represented, namely the first partition of the first hard disk (presumably connected using the first cable and set as master). With UUIDs all this has changed. Apparently for the better, but leaving some folks like me scratching their heads with disbelief.

Yes, Ubuntu is Linux for human beings. Apparently not for all…

PS: For future reference, remember to set the UUID after doing any partition duplication using dd. You do it like this:

tune2fs -U random /dev/sdb1

So you have a LAN with 50+ users and you set up a nice Squid w3cache as a transparent proxy with 100GB of space reserved for the cache (hdds are so cheap nowadays…). Weeks pass and suddenly you notice that something is messing up your web experience as Firefox suddenly decides to run painfully slow. About 30 minutes wasted on finding the culprit (like changing your DNS servers, clearing browser cache, etc.) until you decide to check the router and then the Squid with its logs. And then you find something fishy:

2007/01/01 17:51:19| WARNING! Your cache is running out of filedescriptors
2007/01/01 17:51:35| WARNING! Your cache is running out of filedescriptors
2007/01/01 17:51:51| WARNING! Your cache is running out of filedescriptors
(...)

I won’t be explaining why this happens. Others have done it before. What I’m going to do is present you with a solution that does not require a complete Squid recompilation/reinstallation procedure.

RedHat/Fedora

/etc/init.d/squid stop

nano /etc/squid/squid.conf
  max_filedesc 4096

nano /etc/init.d/squid
  # add this just after the comments (before any script code)
  ulimit -HSn 4096

/etc/init.d/squid start

Debian

nano /etc/defaults/squid
  SQUID_MAXFD=4096

/etc/init.d/squid restart

Ubuntu

nano /etc/default/squid
  SQUID_MAXFD=4096

/etc/init.d/squid restart

And now watch the /var/log/squid/cache.log for a similar line:

2007/01/01 18:32:27 With 4096 file descriptors available

If it still says 1024 file descriptors available (or similarly low value) you are out of luck (or you’ve just messed something up).

This guide is valid for the following distributions:

• 5.04 (Hoary Hedgehog),
• 5.10 (Breezy Badger),
• 6.06 LTS (Dapper Drake),
• 6.10 (Edgy Eft).

While there is a nice tutorial in the Ruby on Rails wiki, it’s by no means complete. According to it, you should only type: apt-get install rails to have the newest Rails installed on Ubuntu. It installs both Ruby and Rails, but what about rubygems? Sorry, not this time. There is also another caveat. Although commands like rails test and ruby script/server are working properly, ruby/console is not. If you had the misfortune of experiencing the aforementioned behavior, then this tutorial is just for you.

Pre requirements:

nano /etc/apt/sources.list

Add the following at the end of the file (replace edgy with breezy if you are running Breezy, dapper for Dapper, etc.):

# All Ubuntu repositories
deb http://archive.ubuntu.com/ubuntu edgy main restricted universe multiverse

Update your apt sources:

apt-get update

Installation:

Install Ruby with developer’s libraries:

apt-get install ruby ri rdoc irb ri1.8 ruby1.8-dev libzlib-ruby zlib1g

Download and install Ruby Gems (no .deb package, unfortunately):

wget http://rubyforge.org/frs/download.php/17190/rubygems-0.9.2.tgz
tar xfvz rubygems-0.9.2.tgz
cd rubygems-0.9.2
ruby setup.rb

Update your RubyGems (also updates the gems cache):

gem update --system

If you get Could not find rubygems-update (> 0) in the repository or a similar error, you need to delete your RubyGems cache:

$ gem env gemdir
PATH_TO_DEFAULT_GEM_REPOSITORY
$ rm PATH_TO_DEFAULT_GEM_REPOSITORY/souce_cache

and

rm $HOME/.gem/source_cache

In the next step install the OpenSSL bindings for Ruby (needed to install signed gems). They are required if you get the following error: SSL is not installed on this system, while installing signed gems like rake:

apt-get install libopenssl-ruby

And the last one:

gem install rails -y

And this is basically it. There are, however, depending on your needs, some…

Additional steps:

One of them is setting up the Rails to connect to the MySQL database in a proper way. We will be using the MySQL C bindings, which, for one, support the MySQL old style passwords (which is set as default for Ubuntu 5.04), but are also significantly faster (in the 2-3x range) than the native Ruby MySQL bindings. First, we will need to install the gcc compiler (and libc6-dev if you don’t have it already installed). Although strange it may seem, as a default it is not installed on a clean Ubuntu installation.

apt-get install gcc libc6-dev

MySQL development libraries are also required (mysql_config plus mysql/include):

apt-get install libmysqlclient14-dev

(for MySQL 5.0 you might be better of with libmysqlclient15-dev).

And now we can install C MySQL bindings:

gem install mysql

If you get "sh: make: not found" do:

apt-get install make

or if you have it already installed, add it to your path:

export PATH=/usr/bin:"${PATH}"

And, of course, in the end install Mongrel:

gem install mongrel -y

And that’s it. Rails installation is complete. Complicated? Not really Happy coding!

After three years (has it been that long?) the time has finally come to replace my very much beloved RedHat 8.0 with something more up-to-date. The choice was fairly simple. Yes, Ubuntu. Praised by everyone, loved by many aka the most popular linux distro around.

I got to know a little bit of Debian magic, by installing apt-get for my RedHat. It had very limited use as the rpm repositories were rarely updated. Only using Debian you can experience its full potential. And yes, Ubuntu is a Debian clone.

The installation was fairly simple, although I missed the “select packages” screen (I suppose there is no option to choose the installed packages before the installation). Next stop: configuration. Turning off graphical login, disabling useless services (cups, alsa, ppp and so on…) – all the usual stuff. Hour after hour my system was shaping up.

At some point I have noticed that I did not have identd running. Without any hesitation I executed apt-get install oidentd. One minute later it was up and running.

And now for the magic part. It has automatically discovered the non-routable IP address (in the 192.168.x.x range) assigned to the eth0 interface (which was also the default routing target) together with the IP address of the gateway. Based on those two IP addresses it found out that my machine was sitting behind a NAT, so it automatically added the -A gateway_ip option to the command line (needed for oidentd to work behind a NAT). That’s just pure magic.

I know it’s just a relatively simple installation script, but still, it’s those simple things which make all the difference. And, sometimes, also make my jaw drop.

10 points for Ubuntu.

One of the magic things about Dreamhost is the ability to login to your account using ssh. Apart from standard things that you can do with it, like compile and install any program that you like (excluding those, which require root access, of course), you can also experience a little bit of magic if you run this command:

tail -f current-httpd-accesslog

There is really something special about tracking your website’s visitors as they come. And, of course, the most interesting thing is actually knowing where they come from (also known as referrers). The problem with this command is that it prints lots of garbage on the screen (like timestamps, response codes, sizes, etc.). The problem is that you cannot grep your real-time tailed log. Why? Don’t ask me, I’m not a Linux guru. You just can’t and that’s it. You can, however, write a script, which deals with that in its own way. And this is what I’ve been writing for the whole day. It was both fun and painful to learn for the n-th time all those shell hacks and quirks. Was it worth it? Sure it was! As a result I came up with this little bash script to trace your referrers in real-time. You can download it or view it below. I must warn you, though. It is highly addictive. Really…

#!/bin/bash

# ========================================================================
# REAL-TIME WEBSITE REFERERS TRACER
# ========================================================================
#
# What?
#   Real-time website referrers tracer is a shell script that lets you
#   trace your visitors as they come. Script works in an ultra compact
#   four-columns view 
#
# Why?
#   Because you cannot do 'tail -f access_log | grep something' and you
#   really want to grep out most of the stuff that your httpd puts in
#   the logs.
#
# Requirements:
#   - website (with not too low and not too high traffic),
#   - shell account on the server where your website is hosted,
#   - access to httpd logs that use the COMBINED format.
#
# Installation:
#   - copy anywhere in your home directory,
#   - edit the script and set the 'log' variable so it actually
#     points to your current httpd log,
#   - make sure the script has execute rights (chmod +x trace-referers).
#
# Running:
#   - just run the script and watch the screen.
#
#
# Version: 0.2 (2005-06-18)
# Author: Paul Goscicki, http://paulgoscicki.com
#
# No copyright rights. You can do whatever you want with this. You may even
# claim this scrip has been written by you from the very beginning 
#
# If you, however, improve it, send me a copy (paul_AT_paulgoscicki.com).
#
# Based heavily on the tgrep script by Ed Morton (morton_at_lsupcaemnt.com):
# http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2004-01/0818.html

# CONFIGURATION
# =============

# Where your httpd log file is
log="current-httpd-accesslog"

# What files to exclude (request for those files won't be shown, regexp syntax)
exclude="\.gif|\.jpg|\.png|\.ico|\.css|\.js"

# Width of request and referrer columns (set it to match your terminal's width)
col_width=35

# MAIN SCRIPT
# ===========

# Check if log file actually exists (and is readable)
if [ ! -r "${log}" ]; then
echo "Cannot access log file: $log"
exit 0
fi

# After startup we will output few lines
start=`wc -l < "${log}"`
start=$(( $start - 30 ))
if (( ${start} < 0 ))
then start=$((0))
fi

# Main loop
while :
do
  end=`wc -l < "${log}"`
  end="${end##* }"
  if (( ${end} > ${start} ))
  then
    start=$(( $start + 1 ))
    sed -n "${start},${end}p" "${log}" | egrep -v "${exclude}" | \
    awk -v col_width=$col_width '{

      # we are only interested in GET/POST requests
      if ( match($0, /\"(GET|POST).*?\"/) > 0 )
      {
        split($0, fields, "\"")

        # IP_ADDRESS
        tmp = $1
        while ( length(tmp) < 15 ) tmp = tmp " "
        printf "%s", tmp " "

        # HTTP_REQUEST (GET/POST)
        tmp = substr(fields[2], 0, index(fields[2], "HTTP/") - 1 )
        tmp = substr(tmp, index(tmp, " ") + 1, col_width)
        while ( length(tmp) < col_width ) tmp = tmp " "
        printf "%s", tmp " "

        # REFERER (the juice)
        tmp = fields[4]
        while ( length(tmp) < col_width ) tmp = tmp " "
        printf "%s", tmp " "

        # USER_AGENT
        printf "%s", fields[6]

        # new line at the end
        printf "\n"
      }
    }'

    start=${end}
  fi

  # this is an endless loop executed every second
  sleep 1
done

Your current hosting provider does not support ssh access? You might then want to read my other post about hosting with dreamhost for as little as 9$/year. Have fun!