Quick start

For those who just want the commands and don’t want to wait:

git clone user@hostname.tld:/path/to/git-svn.git git-svn-clone

cd git-svn-clone

git svn init -s svn://hostname.tld/path/to/svn

git config svn.authorsfile $(basename $(pwd))/git-authors

rsync -avpP user@hostname.tld:/path/to/git-svn.git/svn .git/

git update-ref refs/remotes/trunk origin/master

git svn fetch --all && git svn rebase

Full details

So,  what does all this do? Let’s look at it in stages.

1. Clone the existing git-svn repository, to get the history and avoid hitting the Subversion server any more than we need to (git clone)
2. Set up the Subversion integration information (git svn init)
3. Set up the translation between Subversion authors (bare username) and git authors (name + email address) (git config svn.authorsfile)
4. Copy the Subversion branch metadata from the original git-svn repository
5. Create the local “trunk” reference that git-svn will require
6. Fetch any new revisions from Subversion (and update the revision map between git and Subversion), and then make sure that “master” points at the head of the Subversion trunk

Bonus tip

If you have a number of git branches, it can be a pain to keep rebasing them to the latest trunk (as Subversion doesn’t like non-linear history). This little script may help:

This will automatically rebase all of the branches onto master, although it won’t handle sub-branches very well.

It was still surprising that the bulk of what I wanted to do can be done in about 16 lines of code. That’s right: sixteen.

The plan was to create an edit-in-place system that would degrade nicely in browsers with JavaScript disabled. This isn’t necessarily as elegant as it could be and it doesn’t yet handle all input types, but that’s what version one is for! So without further ado, I present the code:

$.fn.in_place_edit = function() {

  this.each(function() {

    var $eip = $('<span></span>');

    var $el=$(this).hide().after($eip);

    $eip.text($el.val());

    $eip.click(function(){ $eip.hide(); $el.data('origval', $el.val()).show().focus().select(); });

    $el.blur(function(){

      if($el.is('select')) $el.data('origval', $el.val());

      $eip.text($el.data('origval')).show(); $el.hide().val($el.data('origval'));

    });

    $el.keydown(function(e){

      if (e.which==27) $el.val($el.data('origval')).blur();

      if (e.which==13||e.which==9) {

        e.preventDefault();

        $el.data('origval', $el.val()).blur();

      }

    });

  });

};

Usage:

$(function(){ $('input, select').in_place_edit(); });

So, what does all this do? First of all, we create a new function on the main jQuery object, called in_place_edit. This then iterates over elements matching the selector. It hides the input and creates a placeholder <span> element with the input’s value. Then we simply set up event handlers to deal with the placeholder being clicked, the input losing focus, and various keypresses (enter/tab to accept, esc to cancel). To see it in action, I’ve put together a rough-and-ready demo.

Planned future enhancements:

• Checkbox input types (display using the labels?)
• Radio input types
• File input types (if possible)
• Textareas
• Any other types…

Maybe further updates later on!

The host server partitions were LVM-formatted volumes on top of hardware RAID-1, and each of the virtual hosts were partitioned with LVM too, internally. This means that the restore process will not be at all trivial. With just a complete image of the host system, I would need to restore (deep breath) files on a partition on an LVM logical volume (inside a volume group, on a physical volume) in a VMWare hard disk stored on LVM (logical volume inside volume group of physical volumes) inside a disk image that is itself a file on a disk. How very convoluted.

I used a Gentoo system to restore the data, although any Linux system with the appropriate packages should be able to do it. No searches turned up information on doing all of this, and I had to come up with some of it myself, so I thought I would document the process.

Just to illustrate the complexity of the data layout:

Requirements

The machine that is performing the restore requires:

• Linux
• LVM2
• VMWare Server
• losetup
• NBD (Network Block Device) kernel driver
• file system drivers (if applicable)

Procedure

Boot the dead server (vm-host) using a Linux LiveCD and copy its partitions to files on another drive. If you don’t know which partitions the LVM was stored across, you can either copy all of them, or use <tt>pvscan</tt> (possibly after running <tt>vgscan</tt>) to work out which belong to the appropriate volume group. This dump may take some time.

Attach the drive with the partition images to the rescue machine (rescue-host), edit /etc/lvm/lvm.conf on rescue-host, and modify the filter line to allow LVM to scan loopback and network block devices. You’ll probably want to change this back afterwards, so remember what it was.

Set up the partition images as loop devices:

Scan for and activate the volume group:

Find the logical volume with the data we’re after (LogVol06 in volume group VolGroup00 in this case) and mount it:

Copy the VMWare hard disk files out of the LVM mount, because we know that the volume group stored on them will conflict with the existing one:

Check the partition table on the virtual hard disk:

This will give you something like:

The second partition (with type 8E) is the one we wanted in this case, but it’s LVM-formatted. This means we can’t use vmware-mount because an LVM partition cannot be mounted normally.

We need to unmount the currently-active volume group on the loopback devices (i.e. the LVM from vm-host):

Now you need a dedicated terminal to mount the LVM partition from the virtual hard disk, as this process must stay running in order to access the virtual drive:

If you have more than one virtual partition as part of the LVM, you’ll need to have one terminal for each, and select partition numbers and NBD device numbers as appropriate.
Now find the volume group again:

Find the appropriate logical volume and mount it:

Do whatever you need to do with the data, then unmount it all:

And that’s it! I hope this helps someone other than me…

Apache module

The first thing I have in mind is to create an Apache 2.2 module that uses the APR (Apache Portable Runtime) to make configuration file templating easy.

On my server, there are a lot of very similar Apache configuration files. At the moment, they are semi-automatically generated by running a Perl script that reads the configuration from a database and then creates the output files as required. This is all well and good, but changing anything requires running the Perl script, regenerating all of the configuration files, then prodding Apache to reload its configuration. There has to be a better way, especially one that doesn’t require elevated permissions like that.

Enter my proposed solution: mod_sqltemplate. This module will add a few extra settings and (at least) two extra sections to the standard Apache configuration. The propsed features are:

• SQLTemplate_DBDriver «driver» – sets the APR driver to be used
• SQLTemplate_DBOpts «option-string» – options for the driver (e.g. database name, username, password…)
• SQLTemplate_Caching «on|off» – enable/disable configuration caching
• SQLTemplate_CacheDir «path» – directory in which to save cached configurations
• <SQLRepeat «query»> – new section, as described below
• <SQLCatSet «sep» «query»> – new section, as described below

SQLRepeat

The SQLRepeat section is probably going to be the most-used new directive. It provides templating functionality similar to mod_macro, although the template variables are taken from the column names of the query rather than being explicitly specified. The section will then be repeated for each row of the resultset, substitutions occurring as required.

Perhaps an example would help:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  # ...
</VirtualHost>
</SQLRepeat>

For this example, I’ve avoided the configuration directives for clarity, assuming they are specified elsewhere. Supposing the database table contains simply:

Then this will expand to be equivalent to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
  # ...
</VirtualHost>

And so on. These sections can be nested, so if you wanted to store a list of aliases as well:

You could use something like this:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  # ...
  <SQLRepeat "SELECT alias,target FROM apache_aliases WHERE hostid=?" ${id}>
    Alias "${alias}" "${target}"
  </SQLRepeat>
</VirtualHost>
</SQLRepeat>

which would then expand to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
    Alias "/icons" "/var/www/_share/icons"
    Alias "/example" "/var/www/_share/foo"
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
  # ...
    Alias "/bar" "/var/www/_share/foo"
</VirtualHost>

So far, so good. Something to note is that the variable names above did not conflict. If we had included “id” in the list of columns for the inner SQLRepeat, then included ${id}, the module should use the innermost scope. Disambiguation should be obtained by table alias prefixing (e.g. ${apache_hosts.id} or ${apache_aliases.id}). Full variable names should always be specified for clarity. This also shows that if the query returns no results, the repeating section is not included, as you would expect. Also note that queries that use variable substitution should use the prepared statement format, with variables specified afterwards. Variables will be included directly, so anything that might contain spaces must be quoted.

However, there are some directives like ServerAlias that must be specified once with a list of space-separated values. Enter SQLCatSet…

SQLCatSet

This acts in a similar way to SQLRepeat, but it concatenates all of the results for each field before substituting, and it includes its content at most once. If there are no results for the query, then it will not include its content. For example, with a server aliases table like:

You could use something like this:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  <SQLCatSet " " "SELECT alias FROM apache_server_aliases WHERE hostid=?" ${id}>
    ServerAlias ${alias}
  </SQLCatSet>
  # ...
  <SQLRepeat "SELECT alias,target FROM apache_aliases WHERE hostid=?" ${id}>
    Alias "${alias}" "${target}"
  </SQLRepeat>
</VirtualHost>
</SQLRepeat>

which would then expand to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
    Alias "/icons" "/var/www/_share/icons"
    Alias "/example" "/var/www/_share/foo"
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
    ServerAlias test2.example.com test.example.net
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
    ServerAlias *.somewhere.local
  # ...
    Alias "/bar" "/var/www/_share/foo"
</VirtualHost>

This would seem to cover the major functionality needed for automatically generating configurations in a really flexible way, although I’m sure more will probably be needed at some point… like SQLIf, perhaps.

PHP Framework

It occurs to me that having a framework’s code in PHP itself could mean that it is quite slow. It would be interesting to put together a framework in C and embedded as a module, to see if there are any memory/speed improvements possible. It might just be a case of implementing an ORM (Object Relational Model) system to begin with. Obviously, this idea is nowhere near as fully-formed!

The Idea

The basic idea is that we have a sorted list of items that may have some duplicates, say:

{A0, A1, B0, C0, C1, E0, F0, F1, F2, G0, H0, H1}

and we want to remove all of the duplicates, so we get two sets:

{A0, B0, C0, E0, F0, G0, H0} {A1, C1, F1, F2, H1}

Note that it doesn’t matter which of the duplicates ends up in which set, just that the items in the first are all unique and that we don’t lose any. Also, A₀ and A₁ have the same value but the subscript will help to distinguish exactly which of the As I’m talking about. Now we have the problem, let’s look at some solutions.

Lossless solution – additional lists

So the first solution uses two temporary lists, unique and duplicate. We start with two pointers into the original list: p and q. We set p to point to the start of the list, and q to point to the element after p. We then copy the item p points at to our unique list. Then, while we still have more items to examine:

• If the items pointed to by p and q have the same value, then we copy the item q points at to the duplicate list and advance p and q.
• If they are different, we copy the item at q to our unique list, and advance p and q.

We keep doing this until we get to the end of the array, and then copy the unique list to the start of the array, the duplicate list after that (both overwriting the previous contents) and set the number of unique items to the length of the unique

list.

Lossy solution – one list

We can do a similar thing in-place, although this will result in data loss. The essential idea is:

• We start in the same way: p pointing to the first element, and q to the second
• While q has the same value as p, advance q along the list
• When they are different, advance p, copy q to p, and advance q
• Once q goes past the end of the list, we’re done, and the offset of p is the number of unique items.

Now this is quite simple, and only requires one pass through the list, but it results in the loss of information, which is unacceptable in this case.

Lossless solution – one list

For this solution, we can also do it with one pass through the list and two pointers. The basic intuition is that we’re going through the array, accumulating a rotating block of stuff that we don’t want. At any given point:

• Everything between the start and p (not inclusive) is the unique list (so far)
• Everything between p and q (not inclusive) is an unwanted duplicate
• Everything between q (inclusive) and the end is yet to be processed.

At the end, we return the number of unique items at the start of the list; everything above that is a duplicate.

For integers

Here is some pseudo-Java that performs my algorithm for a set of integers.

int set_uniq(int set[]) {

  int count = set.length();

  int tmp, p=0, q=1;

  while (p < set.length && q < set.length) {

    while (p < set.length && set[p] != set[q]) {

      p = p + 1;

      if (q < set.length) {

        if (p < q) {

          tmp = set[p];

          set[p] = set[q];

          set[q] = tmp;

        }

        q = q + 1;

      }

      else if (count > p) {

        count = p;

      }

    }

    while (q < max && set[p] == set[q]) {

      q = q + 1;

    }

  }

  // q hit the end and is still the same as p

  if (count > p)

    count = p + 1;



  return count;

}

The general code

The arguments to the general set_uniq() function are very similar to the arguments to standard qsort():

• The (sorted) array to work on
• The number of items in the array
• The size of each item in the array
• A comparator function that returns a negative integer if the first argument is less than the second, zero if they are equal, and a positive integer if the first argument is greater than the second.

So, without further ado, my code:

int set_uniq(void *set,

    size_t count,

    size_t elem_size,

    int (*cmp)(const void*, const void*)) {

  void *p, *q, *max;

  void *tmp=calloc(1, elem_size); /* TODO: check for failure */

  p=set;

  q=set+elem_size;

  max=set+(count*elem_size);

  while (p< p-set)

      count = (p-set)/elem_size;

    }

    while (q p-set)

    count = 1 + (p-set)/elem_size;

  ifree(tmp);

  return count;

}

Conclusion

This code is released under a Creative Commons licence (see below). A similar idea will work well for other solutions, like set difference. I’ll post those later if anyone really cares. Also, there is a better way of swapping the items by using the fact that:

a ^= b;

b ^= a;

a ^= b;

swaps a and b without needing a temporary variable. If you just go along the data in 32-bit chunks, performing those XOR operations to swap the data. This actually works out to be 30% or so faster (if I remember correctly). Finally – general code like this is always going to be slower than code that’s been specialised for a particular purpose. Calling this on integers, for example, will be much slower than coding an integer-specific version. I should also point out that you can just overwrite integers, as you don’t need to free them