Toolchain


4
Oct 10

DMon status report: evolution to 0.3.7

When I first wrote about DMon, I pointed out that version 0.3 could be already installed from our APT repository, which hosts flavours for i386, amd64 and armel. Five weeks have passed and I feel like I have to write a status update now that 0.3.7 is out. It is not a big version number change because DMon behaves essentially as in 0.3, but a number of new features were added and bugs fixed.

Let’s see which juicy bits are available on top of the features it already had:

  • Additional groups can now be specified using the -u and -U options. As a side effect of their extended syntax, options -g and -G were removed as they are no longer needed.
  • Setting of resource usage limits is now supported with the -r. This uses setrlimit(2) under the hood.
  • DMon can now wait for some time before re-running commands by specifying time intervals with the -i option. Think about it as a poor man’s recurrent job scheduler.
  • Environment variables can now be manipulated (including deletion) by means of the -E option.
  • All commands (dmon, dlog, dslog) can now be built as a single multicall binary. This saves disk space and some kilobytes of memory, which is interestinhg for embedded environments.
  • Tools now honor the DMON_OPTIONS, DLOG_OPTIONS and DSLOG_OPTIONS; environment variables. When defined, they will be interpreted as additional command line arguments. Actual command line arguments take precedence, so values given in environment variablels can still be overriden.
  • A small libnofork.so library is included, which overrides the system-provided fork(2) and daemon(3) functions in such a way that the library can be preloaded (dmon -E LD_PRELOAD=libnofork.so ....) to make programs not to fork. Very useful for applications which create new daemon processes inconditionally so they can be properly monitored. This is tested only under GNU/Linux with glibc at the moment, use with care!

Apart from new features, a couple of bugs related to error handling were fixed, for better reliability.

Last but not least, I would like to thank my friend Andrés J. Díaz who is using DMon and provided feedback and ideas for some of those new features.

Happy monitoring :D


27
Aug 10

DMon: Process monitoring with style

Have you ever wanted to run a lengthy process making sure that it will be restarted on failure? Did you need at some point to daemonize a “normal” program? Well, maybe you already knew about daemontools, runit, freedt, Supervisor, upstart or —recently— systemd. They already do a good job respawning processes, but there are three single things that neither of them are capable of doing:

  • Running a single command and exiting when it exits successfully.
  • Running commands interactively, without daemonizing nor detaching from the terminal.
  • Temporarily suspending execution when the system load goes over a configurable value, and resuming execution automatically as soon as the load drops below another configurable value.

If you ever needed at least one of those features, then DMon is probably what you want. If unsure, read the examples below — you might find some inspiration for use-cases!

The Story

Years ago I discovered Daniel J. Bernstein‘s qmail, and with it came the rest of the “DJB-ware” software stack: djbdns, ucspi, daemontools… and their philosophy!)

Some weeks ago I coded vfand, a small “non-daemon” to control the speed of the fan in my Vaio laptop because it was overheating. I am lazy, so I deliberately left out daemonization and suggested launching it from init(8) — because I knew that DJB’s tools leave daemonization and logging to other tools which just do one thing well.

Days back I had to make a huge data copy in a mission-critical mail server, and used the mighty rsync tool because I wanted the copy to be interruptible so I could stop it when the system load was getting high and then resuming the data copy. I did that manually (Ctrl-Z, wait, fg, repeat), and I do not like performing automatable tasks. Fortunately I seldom do this kind of tasks.

Do you recognize the pattern? DMon is a subproduct of what I have been doing lately, applying the knowledge about daemontools I already had.

How does it work? — Modus operandi

In short it works àla daemontools without control sockets and without using script files for launching processes. All options are specified in the command line, as long as the commands to be run. Like this:

dmon [options] command [command-options]
     [-- log-command [log-command-options]

As an example, consider the following command line:

dmon cron -f -- dlog /var/log/cron.log

This is what DMon wil do:

  1. Daemonize itself.
  2. Create a pipe(2), which will be used to connect the output of the given command to the input of the log-command.
  3. Spawn both the command and the log-command.
  4. Silently wait. If some child is terminated, it will be respawned.

That is pretty close to what the supervise program included in daemontools does, so it have already all the advantages of it, plus without needing stuff in the file system. Passing options to dmon will trigger some of the extra features provided:

  • Passing -n makes it run in the foreground. This is very useful in conjunction with -1: with tha latter the processes will be only respawned if their exit status is non-zero.
  • If you want to log messages from standard error, use -e and both standard output and standard error will be piped to the logging command.
  • For faulty programs which could get somewhat “locked” and sometimes take too much time to run, you may pass a maximum running time with -t. When the timeout is reached the program will be forcibly killed and then started again.
  • Finally, for pausing the program over a given value of system load, use -L. After pausing execution (by means of the SIGSTOP signal), it will be resumed when the system load falls below the value given with -l (by sending SIGCONT). The signals used are the standard ones used for this duty e.g. by the shell, so almost every well-behaved program will work without modifications.

The DMon package already includes a couple tools ready to be used as logging command: dlog will append lines to a log file, optionally adding a timestamp to them, and dsyslog will send lines to the system log. You can use any logging tool which works with daemontools, like multilog (part of it) or my own rotlog ;-)

DMon use-cases & Examples

Running rsync in a terminal (without detaching), pausing the copy when the system load is above 4.0, retrying until the copy succeeds:

dmon -n -1 -L 4.0 rsync -az /path/to/srcdir /path/to/destdir

Launching vfand as a daemon, logging errors to the local syslog, and saving the PID of the process (the second line terminates dmon, vfand and dsyslog in a single shot):

dmon -e -p /var/run/vfand.pid vfand -- dsyslog vfand
kill $(cat /var/run/vfand.pid)

Starting the MediaTomb UPnP server as a user mediatomb (i.e. at bootup), saving auto-rotated logs with rotlog running as user log:

dmon -e -u mediatomb -g mediatomb -U log -G log
     mediatomb --interface eth0 --home /mnt/mediafiles
     -- rotlog -c /var/log/mediatomb/

Final Words

It was fun for me to hack in DMon because C is a language I learnt to love, and using it from time to time is nice to not lose contact with it. Also, I had a clear idea of what I wanted to do for solving a particular problem, which is great for keeping focus.

Albeit DMon is already in its third release (namely version 0.3) and I have been using it since its first inceptions, it may contain bugs as any other piece of software. Do not hesitate to drop me a line with your complaints and suggestions — or even better: get yourself a clone of the Git repository and use its send-email awesomeness!

Have a lot of fun… :-D


21
Jun 10

And even more packages

Most of the time, people involved in the Free Software community what does with packages is using prebuilt packages, and a small amount of people are who actually prepare packages. For one reason or another, I end up building some Debian packages from time to time. Most of the time the motivations are:

  • Having some build-time option enabled (fine-tuning build options via USE-flags is one of the reasons why I love Gentoo :-D ).
  • A new version we want is out but there is no package in Debian stable, nor in backports. Or maybe it is some experimental application we want to use.
  • Add patches in packages which are not yet officially merged, either in Debian or in the upstream project.

In the spirit of giving something back to the community, and because they may be useful to other persons, I like sharing the result of my work, so here we go with a new batch of .deb packages (which most likely will work in Ubuntu as well):

What Distribution Packages
OpenERP client 6.0 alpha 20100617 lenny openerp-client_6.0-alpha20100617_all.deb
DSpam 3.9.0 lenny dspam_3.9.0-4_i386.deb
dspam-doc_3.9.0-4_all.deb
dspam-webfrontend_3.9.0-4_all.deb
libdspam7_3.9.0-4_i386.deb
libdspam7-drv-sqlite3_3.9.0-4_i386.deb
libdspam7-drv-pgsql_3.9.0-4_i386.deb
libdspam7-drv-mysql_3.9.0-4_i386.deb
libdspam7-dev_3.9.0-4_i386.deb
Nginx 0.8.41 lenny / squeeze nginx_0.8.41-0_amd64.deb
nginx_0.8.41-0_armel.deb
rotlog 0.2 lenny / squeeze rotlog_0.2-1_armel.deb

One note regarding the Nginx package: it is supercharged with the versions of the following third-party modules:

The ARM packages are the ones I am using in my GuruPlug, and were built in the plug itself, so I am quite sure they will work in almost any ARM Debian box which uses EABI, including the venerable SheevaPlug and NSLU2 devices.

Have a lot of fun… ;-)


7
Jan 10

Spanish and Galician dictionaries for Vim 7

If you write long, literary text (like end-user documentation) in Spanish and/or Galician, and Vim 7 is your editor of choice, you may want to download the spell checker dictionaries for those languages. To use them, drop the files under ~/.vim/spell. (By the way, you may need to create the directory if needed).

To use the dictionaries, just type in the needed Ex command, e.g. for Galician that would be:

  :set spell spelllang=gl

If you use modelines in your text files, you may want to add those settings there as well. That makes an easy way to choose a different language for each file. Also, do not forget to take a look at the spell checker documentation to learn more about it (tip: some keybindings are really useful).

Note that you will only be able of editing UTF-8 texts. I have not crafted ISO-8859-1 versions of the dictionary tables because no single person should be using an encoding different from UTF-8 nowadays (for a number of good reasons). If someone has a strong need for ISO-encoded tables, please let me know.

Last but not least, let me stress that the dictionaries were converted from the ones used by OpenOffice.org plus some small patches I took from the Vim SVN repository. Big thanks go to the all the people working in both projects. I am not an expert with legal stuff, but as the source files are under the LGPL I think it is safe to assume that the Spanish and Galician dictionaries generated from them are LGPL’d, too.

Remember: it is always good to deliver well-written documents. Happy 2010 ;-)


25
Sep 09

Two months after…

Two months have passed after my last post, and things here have been quieter than usual in this blog, but the world kept moving in the meanwhile so there are some things to tell, and some other that have already been told. Probably the most exciting thing for me and my work mates was the announcement of the shiny Nokia N900. I have resisted the temptation then, because a lot of things have already been said about it. We systems administrators are usually in the shadows, but even so it is a delight working as backing support for people who does big things.

There were a couple of releases this week what are a great example of why Igalia reached its eighth year, and we are still rockin’ in the free world:

  • Hildon 2.2.0, the user interface toolkit used in Maemo. No words are needed to explain how awesome is this!
  • Also, some of my work mates are fine-tuning MAFW after bringing it to life in order to provide multimedia coolness to Maemo, and (who knows!) maybe the desktop as well. Of course all built on top of the lovely GStreamer libraries.
  • GNOME 2.28, which includes the hard work from a lot of work in form all around the world, and a revamped Epiphany web browser which uses WebKitGTK+ as the rendering engine. This means that some of my colleguaes have been killing kitties and their hard work will be deployed in every GNOME install!
  • Frogr 0.1, a tool which carries out out the simple (but important) task of uploading pictures to Flickr, and it is doing a fine task for me right now.
  • In the operating systems ground, Haiku R1 Alpha 1 has been released. This may sound like “some other hobbyist operating system”, but it is a lot more than that: it is a new life for the mighty BeOS R5, which  took eight years to to bring from the dead. I have played a bit with the live CD: the experience is great, although somw rough edges still exist, but I would say that Haiku contains lots of superb work and I am eager to install it in some real hardware.

But more changes apart from software releases happened: in a more personal note, I am now using Fedora 11 in a daily basis at my laptop. Initially I wanted to try out Foresight, because the Conary package manager looks like an interesting piece of software, but unfortunately the installer does not have support for LUKS-encrypted volumes in the installer, and I did not want to bootstrap it manually. Interestingly enough, Fedora does have such support using the same Anaconda installer. I did not like Fedora back in version 6, but I must admit that the community did an impressive improvement (at least comparing version 6 to version 11!) and I am very happy with my current setup. I am even considering Fedora 11 for installing it on my brand new PlayStation 3: I bought one of the old “fat” models, because the new “slim” ones do not officially support installing third-party operating systems.

Finally, a quick note to finish this “I am alive” post: I am glad that we have decided to push Linux-vServer in our servers, because we are getting some interesting benefits thanks to it, being the main one the ability to easily clone a running machine and use the clone for testing purposes before applying changes in the production environment. Also, we are now able of easily provide sandboxed environments in which users have almost-full administrative privileges without having to worry about other services being affected in case something goes wrong. And we are getting those niceties with a minimal overhead (~1.5%) in terms of kernel CPU usage. As we are moving services which were previously run on physical machines into virtual machines, we are saving power and contributing to the environment while providing a better service and support to our staff :-D


8
Jan 09

Get Marked

If you are a Vim user (like me) you may be surprised to know (as I was today) that you can add jump marks to save positions in files and then go back to the marked positions used simple keystrokes:

  • With mX, where X is a letter, a mark will be set.
  • With 'X, Vim will take you to the line marked with mark X.

Easy, isn’t it? If you are curious, you can read more on this Linux.com article. And do not forget to configure a viminfo file to save marks between edit sessions and you will get the full mark-on-the-go feature pack :-D


20
Nov 08

Convert repositories with fast-{export,import}

Update 2010-01-23: Nowadays the plugin is packaged in Debian (package bzr-fastimport) and Ubuntu (package bzr-fastexport). Once installed you can just go and use bzr fast-export instead of the alias. Thanks to the people that commented about this update :)

Those days I faced the need of moving a set of Bazaar branches into a set of Git repository, and althought I already knew about Tailor, but I wanted a one-shot conversion and Tailor is more, ahem, tailored for periodic synchronization. So I decided to try git fast-import, because I already knew about bzr fast-import plugin which is inspired by the Git version of the thing. The Bazaar plugin includes a bzr-fast-export script which can be used to feed input data to git fast-import.

First, we need to install the Bazaar plugin. This one is easy:

mkdir -p ~/.bazaar/plugins
cd ~/.bazaar/plugins
bzr get lp:bzr-fastimport fastimport

I will be importing three branches into a Git repository, so we need to create a Git repository:

mkdir ~/myproject.git
cd ~/myproject.git
git --bare init

I suppose you have three related Bazaar branches:

  • ~/myproject is the main branch of the project. It will be imported as master branch in Git.
  • ~/myproject.branch-a and ~/myproject.branch-b are two branched whose parent is the first one.
alias bzrexp="~/.bazaar/plugins/fastimport/exporters/bzr-fast-export"
bzrexp --export-marks=../marks.bzr ../myproject
   | git fast-import --export-marks=../marks.git
bzrexp --marks=../marks.bzr --git-branch=branch-a
   ../myproject.branch-a | git-fast-import
   --import-marks=../marks.git --export-marks=../marks.git
bzrexp --marks=../marks.bzr --git-branch=branch-b
   ../myproject.branch-b | git-fast-import
  --import-marks=../marks.git --export-marks=../marks.git

Voilà! It will run in a snap, the limit will be your disk speed, as both the exporter and the importer are very fast.


5
Nov 08

Using Bleeding-edge Sup on Ubuntu

This is a quick recipe which will tell you how to run the development version of the Sup e-mail client on Ubuntu 8.10; the method will probably work on any recent Ubuntu or Debian distribution.

First, download the latest Sup code from their Git repository:

cd ~
git clone git://gitorious.org/sup/mainline.git sup

Then, install dependencies needed for Sup to run, as root:

apt-get install rubygems ruby-dev
   lib{ferret,highline,openssl,net-ssh,ncurses,gettext}-ruby1.8
gem install hoe rmail mime-types fastthread
   trollop lockfile chronic

Now you can run Sup with your regular user account from the Git branch you downloaded. Do not add sources with sup-config as it will fail when trying to find the sup-add command, it is better to run sup-add manually like in the following example:

cd ~/sup
ruby -I lib -w bin/sup-config
ruby -I lib -w bin/sup-add --labels=inbox imaps://igalia.com/
ruby -I lib -w bin/sup

Of course, make sure you use your own mail server, and run sup-add --help to take a look at the possible options. If you have already configured Sup before, or if your prefer to, you can skip running sup-config and/or sup-add.

That’s all for now, folks. I will tell you in the following weeks how well I manage using Sup. It can be an interesting reading, as I am trying to move away from Evolution. As a last reminder, do not forget to take a look at the Sup new user guide :-)

Update: You will need to install build-essential and libncurses5-dev if you do not have them installed prior to installing Ruby gems.


15
Sep 08

Beware of bugs… yarrr!

Beware of bugs in the above code; I have only proven it correct,
not tried it.

—Donald E. Knuth.

As long as you keep administering machines, you may have found that updating critical systems can be tedious if you do your job carefully: manually checking whether packages (either new or upgraded) have known issues which may affect your machines can be tedious and error prone. Fortunately there is some tools which automate such tests, and one of those is available for Debian and derived systems (in general I think it will work with nearly every APT-based distro out there): apt-listbugs. After installing the package, which is readily available from your Usual Sources™, the installed APT hook will check the package against the bug tracker for your distribution and ask you what to do in the presence of existing open bugs:

Retrieving bug reports... Done
Parsing Found/Fixed information... Done
serious bugs of python-babel (-> 0.9.1-5)
 #493742 - Python namespace conflict with python-pybabel
Summary:
 python-babel(1 bug)
Are you sure you want to install/upgrade the above packages? [Y/n/?/...]

You can tap ? at the prompt to see the list of available actions: you can stop installing, pin packages… If you also install reportbug you will be able of typing the bug number at the prompt to see bug details.

I hope this makes you life a little bit happier :)


26
Jun 08

Tip: Faster remote shells with multiplexed SSH

Edit ~/.ssh/config (or /etc/ssh/ssh_config for sitewide configuration) and add the following snippet:

Host *
   ControlMaster auto
   ControlPath ~/.ssh/socket-%r@%h:%p

This will create a socket for each set (user, machine, port) when the first SSH session is opened. Further sessions will see the socket and use it instead of opening a new connection, multiplexing all concurrent connections via the same connection. The same goes for scp and sftp.

Nice side-effects of this:

  • No functionality is lost at all.
  • SSH sessions will open faster, as there is no need to establish a connection.
  • You will not need to enter you password everytime (but note that maybe you should be using public-key authentication).
  • You can open several sessions to servers which put a limit on the number of simultaneous connections.
  • If you are a sysadmin, you can limit the number of SSH connections to exactly one per user :-)

Hope you find this tip useful :-P


Bad Behavior has blocked 262 access attempts in the last 7 days.